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FOREWORD 

The original edition of the Procedures for ASHRAE Standards Actions (PASA), dated June 30, 1994 superseded 
all previous documentation for communicating ASHRAE’s procedures as a basis for continuation (re¬ 
accreditation) under the ANSI Organization Accreditation Method. PASA changes must be approved by the 
ASHRAE Board of Directors and ANSI. 

ASHRAE publishes the following types of voluntary consensus standards: 

ASHRAE Standard Method of Measurement or Test 
ASHRAE Standard Design 
ASHRAE Standard Practice 
ASHRAE Standard Rating 

Most ASHRAE Standards are of the Method of Measurement or Test type. ASHRAE Standard Design and 
Standard Practice documents receive the most use by consulting engineers and architects, requests for 
committee participation, public review comments, and adoption by code bodies. HVAC equipment 
manufacturers use all three types of ASHRAE Standards. The project committee voting memberships represent 
a balance of interest (at least User, Producer, and General) so that no one category has a majority. ASHRAE 
Standards are used by persons in all three-interest categories. 

ASHRAE’s Standard Project Committees may include persons who are not members of ASHRAE (e.g., 
physiologists, medical doctors, chemists, etc.). 

The Summary of changes table has been moved to the end of the document. 
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PROCEDURES FOR ASHRAE STANDARDS ACTIONS 


1 INTRODUCTION 

Founded in 1894, the American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. 
(ASHRAE) is a technical society of more than 50,000 members, organized and operated for the exclusive 
purpose of advancing the arts and sciences of heating, refrigeration, air conditioning and ventilation, the allied 
arts and sciences, and related human factors for the benefit of the general public. ASHRAE sponsors a research 
program, develops standards, publishes technical data, and organizes meetings and educational activities for 
both its members and others professionally concerned with refrigeration processes and the design and 
maintenance of indoor environments. The Society also strives to promote increased public awareness of the 
requirements for healthful and comfortable indoor enviro nm ents. 

2 SCOPE 

These Procedures direct ASHRAE’s standards activities in the field of heating, refrigeration, air conditioning 
and ventilation, and the allied arts and sciences. These Procedures apply to activities related to the development 
of consensus for approval, revision, reaffirmation, withdrawal, and maintenance of ASHRAE Standards, and to 
relations with standards-related committees of other organizations. 

ASHRAE leaves to trade associations the writing of rating standards unless a suitable rating standard will not 
otherwise be available. 

3 DEFINITIONS, ABBREVIATIONS AND ACRONYMS, AND CLASSIFICATIONS 
Annex A provides definitions, abbreviations and acronyms, and classifications of ASHRAE Standards. 

4 APPROVAL OF PROPOSED STANDARDS 
4.1 RESPONSIBILITY 

The Standards Committee is responsible for formation of project committees and the development, preparation, 
interpretation, revision, reaffirmation, withdrawal - and submittal to the Board of Directors or its designee for 
approval - of ASHRAE Standards Actions. The Board of Directors or its designee will counsel and offer 
guidance to the Standards Committee on policy level standards. 

Each member of the Standards Committee is appointed to one or more subcommittees by the chair. These 
subcommittees are responsible for: 

■ tracking the status of project committees, 

■ recommending ASHRAE appointments to standards-writing committees of other organizations, 
monitoring their activities, and maintaining ASHRAE participation in the canvass balloting activities of 
other standards-writing organizations, and 

■ ensuring the timely maintenance of existing standards in accordance with ASHRAE procedures; 
forming interpretations committees for standards when project committees do not exist; considering 
requests for development of joint sponsorship agreements; and acting in coordination with cognizant 
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ASHRAE Technical Committees, Task Groups or Technical Resource Groups (TC/TG/TRG) to 
recommend reaffirmation or withdrawal of standards. 

Project Committees are appointed to develop and revise standards in accordance with approved written 
procedures. The project committees are responsible for the technical content of standards and addenda. The 
Standards Committee supervises the work of project committees to ensure that approved procedures have been 
followed. 

4.2 STANDARDS COMMITTEE MEMBERSHIP 

4.2.1 Standards Committee 

The Standards Committee is a standing general committee and its members are elected by the Board of 
Directors. The members are selected from various interest groups to prevent dominance of any single interest 
and may include persons from groups such as manufacturers, consultants, educators, trade associations, 
government, testing/research laboratories, utilities, code bodies, contractors, consumer/users, and 
environmentalists. Members of the Standards Committee must be of Fellow, Member, or Associate Member 
grade. Members of Standards Committee may be Life Members or Presidential Members. 

4.2.2 Standards Committee Subcommittees 

The Standards Committee has the following subcommittees: a) the International Standards Advisory 
Subcommittee (ISAS), b) the Intersociety Liaison Subcommittee (ILS) c) the Planning, Policy and 
Interpretations Subcommittees (PPIS), d) the Standards Project Liaison Subcom mi ttee (SPLS), e) the Code 
Interaction Subcommittee (CIS), and f) the Standards Reaffirmation Subcommittee.(SRS) 

4.2.2.1 International Standards Advisory Subcommittee (ISAS) 

ISAS is responsible for monitoring, reporting and submitting recommendations to the Intersociety Liaison 
Subcommittee concerning ASHRAE’s regional and international standards activities. ISAS is comprised of 
StdC and non-StdC members with knowledge of International Standards Development. 

4.2.2.2 Intersociety Liaison Subcommittee (ILS) 

ILS oversees the Society’s participation in the standards work of other standards development organizations, the 
American National Standards Institute (ANSI), and ANSI’s Technical Advisory Groups on ISO and IEC 
standards. ILS is comprised of StdC members only. 

4.2.2.3 Planning, Policy and Interpretations Subcommittee (PPIS) 

PPIS oversees the maintenance and revision of all standards writing and processing procedures and policies, 
recommending approvals of new Titles Purposes and Scopes and handling interpretations of existing standards 
when no project committee exists and evaluates requests for joint sponsorships of SCDs. PPIS is comprised of 
StdC members only. 

4.2.2.4 Standards Project Liaison Subcommittee (SPLS) 

SPLS oversees the development of standards committee documents (SCDs), training of PC Chairs, oversees 
work plans, and waivers of the ASHRAE Units policy. SPLS is comprised of StdC members only. 

4.2.2.5 Code Interaction Subcommittee (CIS) 

CIS oversees the participation by ASHRAE in the development of model codes and standards by other SDOs 
that have relevance to ASHRAE technical interests. CIS is comprised of StdC and non-StdC members with 
knowledge of model code development and the deployment of building regulations. 

4.2.2.6 Standards Reaffirmation Subcommittee (SRS) 



PASA - Procedures for ASHRAE Standards Actions 
ANSI Approved: April 29,2015 


JA1786 



Case l:13-cv-01215-TSC Document 122-8 Filed 12/22/15 Page 108 of 174 
USCA Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 36 of 573 

SRS serves as the project committee (consensus body) for reaffirmation, withdrawal or revision (when updating 
references will not make a substantive change to the standard or guideline) of existing ASHRAE standards. 

SRS is a project committee of at least five (5) members, including at least three members of the StdC and 
applicants responding to a call for members posted in ASHRAE Standards Actions. SRS acts, in limited 
circumstances, as a project committee for existing standards and is subject to the rules of project committees for 
reaffirmations, withdrawals, and revisions only to update references, that are not themselves reaffirmations and 
do not cause a substantive change to the standard. SRS must comply with all ANSI requirements for openness, 
balance and due process. SRS may act in lieu of a PC, with the advice of the cognizant TC/TG/TRG, to 
recommend, reaffirm, withdraw or revise an existing standard based on updated references (that do not cause a 
substantive change to the standards) or add a second system of units to an existing standard, thereby making the 
existing standard useable in either SI or IP units. (See Standards Action Annex A .) 

4,3 ESTABLISHMENT OF PROJECT COMMITTEES 

4.3.1 Project Committees 

Project committees are authorized by the Standards Committee as either Standard Project Committees (SPCs), 
which are ad hoc committees, or Standing Standard Project Committees (SSPCs). Project committees are the 
consensus-forming bodies of the Society and no single interest may have a majority vote unless waived in 
writing (including electronic communication) by the other interests (see balance, Annex A) . Efforts to recruit 
materially affected and interested parties from diverse interest categories to become members of a non-balanced 
SPC shall be on-going and documented. 

A member of the SPLS is appointed as StdC Liaison to the new project committee. A call-for-members 
a nn ouncement is conducted. Drawing from the resulting applications and recruiting efforts, candidate 
committee members are recommended in consideration of their personal expertise and their effect on committee 
balance. Recommended members and non-policy level PC Chairs are approved by a majority vote of a 
designated subcommittee of Standards Committee, normally SPLS. Standards Committee must concur by 
majority vote for all policy level PC Chairs. 

4.3.2 Project Committee Voting Status 

Project Committees may have project committee voting members (PCVM), non-voting members (NVM), 
project subcommittee voting members (PSVM), or consultants. 

4.3.3 PC Activity Initiation 

At the first official business meeting of a new PC, the PC shall vote on whether to concur with, or propose 
changes to, the original TPS. The PC may conduct business (for example, pass motions) only after the 
membership roster with at least 5 voting members has been approved by SPLS or the StdC. However, the PC 
Chair may hold organizational meetings for individuals interested in becoming members of the PC, and the 
group may begin developing the standard or guideline. 

4.3.4 Use of Subcommittees 

The PC Chair may organize the committee structure using formal subcommittees. If subcommittees are used, 
the Chair’s recommendation for subcommittee Chair must be approved by SPLS. Responsibilities of various PC 
subcommittees typically are to develop drafts of one or more assigned clauses of a standard, annexes, or 
addenda; prepare a system of units; prepare text in appropriate language; establish educational activities; 
develop draft responses to requests for interpretation; or develop proposed responses to comments resulting 
from public review. Subcommittee actions shall be submitted as recommendations for action by the parent PC. 
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4.3.5 Project Committee Officers 

PC officers consist of a Chair, Secretary, and in some cases also Vice Chair(s) and Subcommittee Chair(s). The 
Chair and any Vice Chairs or Subcommittee Chairs must be ASHRAE members. Only individual members as 
defined in Section 4.3.6 are eligible to serve as Chair, Vice Chair or a Subcommittee Chair. The Chair shall 
appoint a Secretary and recommend a Vice Chair, if the size or activity of the PC warrants one. 

4.3.6 PC Members 

A PC shall have individual members and designated PCs may have organizational members. Individual 
members are appointed as “personal members,” not as representatives of any organization, corporation, 
partnership, or employer. An organizational member designates a representative, and at the organization’s 
discretion, an alternate, to serve in the absence of the representative, to participate in PC activities in the same 
ma nn er as an individual member, except that the representative and alternate may not serve as a Chair or Vice 
Chair of a committee in accordance with 4.3.10. There shall not be more than one PCVM from any one 
company, association, agency, or entity. 

4.3.7 Participation in Committee Activities 

Each PC member is expected to attend meetings and participate in other committee activities, such as 
conference calls, letter ballots, e-mail correspondence, etc. Failure to regularly do so, without an acceptable 
reason, shall be sufficient cause for the PC Chair to recommend to SPLS removal of a person from the PC 
membership roster. 

4.3.8 Removal for Cause 

The PC Chair may recommend removal of a PC member from the roster for due cause, by submitting a 
recommendation and justification in writing to the SPLS Liaison and Manager of Standards (MOS). PC Chair 
recommendations for termination of the membership can be based on a failure to actively participate in the PC 
proceedings or meet PC responsibilities, including but not limited to: missing two consecutive PC meetings 
without prior written approval from the PC Chair; failure to attend at least 50% of scheduled PC meetings within 
any twelve month period; and/or failure to return at least 60% of the letter ballots within any twelve month 
period. The MOS will transmit the reco mm endations of the PC Chair and SPLS Liaison and related 
correspondence to SPLS for action in a meeting or by letter ballot. The SPLS Chair may call an executive 
session of the SPLS or the PC to discuss the matter. Failure to fully disclose any conflict of interest shall be 
grounds for removal from the PC. 

4.3.9 Removal for Cause Initiated by SPLS 

SPLS may, without a recommendation of the PC Chair, recommend removal of one or more PC members from 
the roster for any of the reasons stated in 4.3.8. SPLS may also recommend removal of a PC member from the 
roster of one or more PCs due to a conflict of interest (defined in Ann ex A) or a violation of the ASHRAE Code 
of Ethics by submitting a recommendation and justification in writing to the MOS. 

4.3.10 Organizational Members 

Subject to approval of SPLS, the PC Chair may nominate an organization as an organizational member (OM). 
The designated organizational representative (OR) of the OM may serve as a PCVM or a PSVM of the PC. For 
consideration of appointment as an OM, the organization should normally be a governmental agency, public 
interest group, or organization that represents a number of entities such as a trade association. Organizations 
such as educational institutions or corporations and partnerships engaged in commerce shall not be eligible for 
OM status. 

Organizations are informed of the availability of organizational memberships on specific PCs by one or more of 
the following: 

a) notice in ASHRAE Insights, ASHRAE Journal, ANSI Standards Action, etc.; 
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b) posting on the ASHRAE Web Site; 

c) press releases to the applicable trade press; or 

d) direct communication to potential materially-affected organizations. 

4.3.11 Criteria for Considering Organizational Members 

The PC Chair should consider the following criteria in nominating organizations for OM status on a PC: 

a) the degree to which members of the organization are materially affected by the requirements of the 
standard; 

b) the ability of the representative of the organization to represent the interests of the members of the 
organization; 

c) the capability of the organization to provide an individual with appropriate technical or scientific 
qualifications to serve as their representative, and if desired, another individual with appropriate 
technical or scientific qualifications to serve as an alternate organizational representative (AOR); 

d) that an official representative of the organization has endorsed the member and the alternate to serve on 
the project committee; and 

e) the willingness of the organization to abide with the term s of organizational membership. 

4.4 Project Committee Size 

The PC shall consist of no less than 5 PCVMs with no upper limit, including the Chair. In addition to the 

PCVMs, the PC membership may also include PSVMs if the PC is organized into subcommittees or NVMs if 

not organized into subcommittees. 


5 RELATIONSHIPS WITH OTHER STANDARDS-DEVELOPING ORGANIZATIONS 

5.1 General 

The Standards Committee supervises ASHRAE’s participation in the standards work of other organizations 
including the American National Standards Institute (ANSI) and international and regional standards 
organizations including the International Organization for Standardization (ISO) and the International 
Electrotechnical Commission (IEC). 

5.2 Joint Sponsorship 

A request to jointly sponsor a standard shall be evaluated by the Standards Committee, considering overlap of 
expertise and responsibility. The evaluation must be reported to Technology Council. A recommendation for 
joint sponsorship including a recommendation for the lead organization shall be forwarded to the Technology 
Council and Board of Directors for approval. A recommendation against joint sponsorship shall be forwarded 
as an information item to the Board of Directors. If joint sponsorship is approved by the Board of Directors, 
standards-writing and approval procedures must be negotiated with the other organization by the MOS on behalf 
of the Standards Committee. 

The standards-writing and approval procedures should be those of the lead organizations. If ASHRAE 
procedures are not adopted, the adopted procedures must be compatible with ASHRAE procedures in regard to 
openness of proceedings, public review of drafts, and delegation of technical content to the project committee. 

6 COMPLIANCE WITH AMERICAN NATIONAL STANDARDS INSTITUTE (ANSI) 
REQUIREMENTS FOR ACCREDITATION 

Since 1976, ASHRAE has been accredited by ANSI as a developer of American National Standards and 
continuation of this accreditation shall be maintained based on ASHRAE procedures and practices for standards 
development meeting the criteria for accreditation given in ANSI Essential Requirements: Due process 
requirements for Am erican National Standards (referenced hereafter as ANSI-Essential Requirements). 
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7 CRITERIA FOR APPROVAL, WITHDRAWAL, AND DISCONTINUANCE OF ASHRAE 

STANDARDS 

7.1 INTRODUCTION 

Approval of an ASHRAE Standard requires verification that the requirements for due process and consensus 
have been met. Approval thus ensures that each ASHRAE Standard is generally acceptable to the directly and 
materially affected interests. 

7.2 GENERAL 

Standards shall be designated, developed, published, and maintained in accordance with these Procedures. 

7.2.1 Public Review 

7.2.1.1 Advisory Public Review (APR) 

A PC may vote by majority of the voting membership to recommend to the SPLS Liaison and SPLS Chair that a 
draft SCD, or portion thereof, be subjected to an APR if the PC believes that the draft contains new, unusual or 
potentially controversial elements that the PC believes would benefit from increased public scrutiny prior to 
finalizing the draft for publication public review (no continuation letter ballot, no roll call vote record, no 
marked up roster, or submittal form is needed). Any comments received as a result of an APR are deemed to be 
"supportive" and do not need to be "resolved". Apart from acknowledging receipt of each comment, 
communication with the commenters is optional but may be undertaken to clarify a comment's intent or to invite 
further participation in the standard development process. The underlying concept of the APR is to gain 
increased public participation early in the development process and thus to deal with, and potentially resolve, 
controversy before publication approval is sought. APRs are not submitted through the ANSI process. 

7.2.1.2 Normal Track Public Review (NTPR) 

A standards action approved by the PC for publication public review that meet any of the following criteria 
shall be processed as a normal track: 

a) there are negative votes with reason within the PC; 

b) a credible threat of legal action (in writing) against ASHRAE has been made related to the proposed 
draft; 

c) the proposed draft is related to a Policy Level Standard ; and 

d) the SPLS Liaison has notified the MOS within ten calendar days, from the receipt of the package, with 
specific justification, that the PC has violated due process. 

SPLS must approve the SCD before it can be issued for public review. 

7.2.1.3 Fast Track Public Review (FTPR) 

A standards action approved by the PC for publication public review that meet all of the following criteria shall 
be processed as a fast track: 

a) there are no negative votes within the PC; 

b) no credible threat of legal action (in writing) against ASHRAE has been made related to the proposed 
draft; 

c) the proposed draft is not related to a Policy Level Standard (Policy Level PC Chair may request an 
exception. The SPLS Chair must grant or deny the exception within ten working days of submittal); and 

d) the SPLS Liaison has not notified the MOS within ten calendar days, from the receipt of the package, 
with specific justification, that the PC has violated due process. 

No additional approvals for issuing the SCD for public review are required. 
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7.2.2 Publication Approval 

Approval of Standards Action by the ASHRAE Board of Directors that have unresolved objectors (commenters 
or negative PC votes with reason) or a threat of legal action shall be preceded by formally voted 
recommendations by the project committee and Standards Committee. 

Approval of Standards Actions by Technology Council that are policy level SCDs that have no unresolved 
objectors and no threat of legal action shall be preceded by formally voted recommendations by the project 
committee and Standards Committee. These Standards Actions shall be reported as an information item to the 
ASHRAE Board of Directors. 

Approval of Standards Actions that are not policy level, that have no unresolved objectors and no threat of legal 
action shall be preceded by formally voted recommendations by the project committee and processed for 
publication by ASHRAE Staff. These Standards Actions shall be reported as an information item to the 
Standards Committee and the ASHRAE Board of Directors. 

The SCD shall be deemed to have been approved by the BOD upon approval of its designee. 

7.2.3 Quorum Requirements 

To conduct standards-related business at a meeting of a project committee, StdC or its subcommittees, 
Technology Council or the Board of Directors, a quorum must be present. A quorum exists if a majority of the 
voting membership is present. 

7.2.4 Voting Requirements for Standards Actions 

Standards actions recommendations must be approved by the project committee (consensus body) with (1) 
affirmative recorded votes by the majority of the membership of the project committee and (2) affirmative votes 
from at least two-thirds of those voting, excluding abstentions of the project committee. When recorded votes 
are taken at meetings, project committee members who are absent shall be given the opportunity to vote before 
or after the meeting. Persons who cast negative votes on a standards action shall be requested to comment on 
reasons for their negative votes. If the vote passes with one or more negative votes with reasons for those 
negative votes, the results shall be held in abeyance until the comments and attempts at resolution of comments 
(including those unresolved comments received in response to the formal ASHRAE public review (See 
Section7.4.6) are transmitted to all eligible voters and they are given an opportunity to change their vote, 
reaffirm their vote, or to vote. A written response to negative voters with reason voting at a meeting or via letter 
ballot shall be issued advising each of the disposition of the objection and the reasons why. 

Standards Committee, Technology Council and the Board of Directors recommendations for standards actions 
must be approved by a majority of those voting at a meeting of the Standards Committee, and Board of 
Directors, or by letter ballot. 

7.2.5 Voting Rules for Letter Ballots By Project Committees 

The Chair of the PC (or its subcommittees) may authorize a letter ballot to be issued on any matter. Actions of 
the PC and subcommittees conducted by letter ballot require approval by a majority of the voting membership of 
the committee. Standards actions, and issuance or revision of an official interpretation require affirmative votes 
of the majority of the membership and of at least two-thirds of those voting, excluding abstentions. When a 
letter ballot is conducted via e-mail it is intended that members will not use “Reply to All,” but reply only to the 
sender of the e-mail. A written response to objectors on a letter ballot vote shall be issued, advising each of the 
disposition of the objection and the reasons why. 

7.2.6 Negative Votes on Letter Ballots of PCs and Project Subcommittees 
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Persons who cast negative votes on a letter ballot shall be asked if they wish to comment on reasons for their 
negative votes. If the vote passes with one or more negative votes, the results shall be held in abeyance until the 
comments are transmitted to all eligible voters and they are given an opportunity to reaffirm their vote, change 
their vote or to vote (by letter ballot or at the next meeting). If a reason is not provided for a negative vote, the 
eligible voters are informed of the negative vote by distribution of the letter ballot results. 

The Chair of the entity voting by letter ballot may offer rebuttal to the comments of the negative voters. After 
the eligible voters have had ample opportunity (not in excess of two weeks if by letter ballot) to reaffirm their 
votes, change their votes or to the vote , the results shall be final. If negative votes with comments are received 
on the second round, all eligible voters will be informed but no further opportunities to change votes will occur. 

7.3 MAINTENANCE OF STANDARDS 

ASHRAE Standards shall be maintained under periodic maintenance procedures except when use of continuous 
maintenance procedures has been voted by the Standards Committee. (See definitions of continuous 
maintenance and periodic maintenance in Annex. A .) 

When a PC does not exist, a designated subcommittee of StdC shall (a) form Interpretation Committees to 
respond to requests for interpretation, and (b) with the advice of the cognizant Technical Committee, Task 
Group, or Technical Resource Group, shall provide recommendations to the Standards Committee concerning 
the need for reaffirmation, revision based on updated references or adding a second system of units to a 
standard, thereby making the standard useable in either SI or IP units, withdrawal or the need to form a new 
project committee to revise a standard. (See TC, TG, and TRG, Annex. A .) 

7.4 DUE PROCESS REQUIREMENTS 

The following represent the due process requirements for development of consensus. 

7.4.1 Openness 

7.4.1.1 Access 

Meetings of the Standards Committee, PCs, and their subcommittees are open to all members of ASHRAE and 
to members of the public who are directly and materially affected by ASHRAE’s standards activities. When 
there is a discussion of a sensitive issue or of a personal nature, the chair of any of these committees or 
subcommittees may declare an Executive Session, during which only members of the committee or 
subcommittee and such other individuals invited by the chair shall be present. 

7.4.1.2 Barriers 

There shall be no undue financial barriers to participation in project committees. Participation shall not be 
conditional upon membership in ASHRAE or in any standard cosponsoring organization, or unreasonably 
restricted on the basis of technical qualifications or other such requirements. (See due process in Annex A .) 

7.4.1.3 Notice 

Timely and adequate notice of the initiation and development of a new standard or a substantively revised 
standard and the establis hm ent of a new PC shall be on the ASHRAE web site. In addition, proposals for new 
American National Standards and proposals to revise, reaffirm, or withdraw approval of existing Am erican 
National Standards shall be transmitted to ANSI for listing in Standards Action. Notices should include a clear 
and meaningful description of the purpose of the proposed activity. 
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7.4.2 Lack of Dominance 

The standards development process shall not be dominated by any single interest category, individual or 
organization. Dominance means a position or exercise of dominant authority, leadership, or influence by reason of 
superior leverage, strength, or representation to the exclusion of fair and equitable consideration of other viewpoints. 
Unless a claim of dominance is submitted in writing (electronic communications) by a directly and materially 
affected party, no test for dominance is required. (See Section 7.4.3, and balance, dominance, and interest 
category in Annex A .) 

7.4.3 Balance and Interest Categories 

Historically the criteria for balance are that a) no single interest category constitutes more than one-third of the 
membership of a consensus body dealing with safety-related standards or b) no single interest category 
constitutes a majority of the membership of a consensus body dealing with other than safety-related standards. 

The interest categories appropriate to the development of consensus for a standard are a function of the nature of 
the standard being developed. In defining the interest categories appropriate to the standards activity, 
consideration shall be given at least to the following: 

Producer 

User 

General 

Where appropriate, more detailed categories or subcategories may be considered. 

7.4.4 Additional Procedures 

ASHRAE shall, as deemed appropriate and needed, provide additional forms, commentary, examples, 
educational materials, and related information that will support the application and use of these procedures. 

7.4.4.1 Appeals to BOD 

Annex B provides an appeal mechanism for procedural complaints regarding any BOD action or inaction. 

7.4.4.2 Complaints of Inactions by the Standards Committee, its Subcommittees or Project Committees 

In addition to formal appeal of Board standards actions or inactions, failure of the Standards Committee, its 
subcommittee(s), or a Project Committee to consider a written request may be addressed by writing (including 
electronic communication) to the Manager of Standards at any time. (See Annex D .) 

7.4.5 Public Review Period 

The public review comment period shall normally be the minimum allowed by ANSI unless more time is 
justified. Limited revisions (ISCs) and addenda up to 5 pages may have a 30 day comment period. 

7.4.6 Consideration of Public Review Comments Received 

All comments to public review drafts shall be submitted electronically via the online comment database. An 
exception to this rule may be granted by the MOS if the commenter can demonstrate that he/she does not have 
ready access to the internet. The PC Chair or his/her designee shall submit responses to commenters 
electronically in the medium specified by MOS. 

Public Review Comments received during open public review shall be reported to all members of the PC. 

Prompt consideration shall be given to all public review comments, including those received through ANSI. An 
effort to resolve all negative public review comments shall be made, and each negative commenter shall be 
advised in writing (including electronic communication) of the disposition of the objections and reasons there 
for. (See substantive change in Annex A .) After consideration of comments or because of new information 
received, the PC may make changes to the draft. Any substantive changes in the draft must be approved and 
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voted on by the PC for publication public review. The PC may consider any public review comments received 
after the close of the public review period, or shall consider them as a new proposal. 

7.4.6.1 Late Comments Received Under Periodic Maintenance 

Comments received after close of open public review under ASHRAE’s periodic maintenance procedures may 
be held for consideration at the next revision at the discretion of the PC. 

7.4.6.2 Comments Received Under Continuous Maintenance 

An SSPC that is designated by the Standards Committee as operating under continuous maintenance procedures 
shall take documented, consensus action on each request for change to any part of its standard. 

7.4.7 Consideration of Standards Proposals 

Prompt consideration shall be given by the Standards Committee to proposals made for developing new 
standards or revising, reaffirming, or withdrawing existing standards. 

7.4.8 Records 

Records shall be maintained to provide evidence of compliance with the record retention policy in the ANSI 
Procedures. Records concerning new, revised, or reaffirmed periodic maintenance standards shall be retained 
for one complete standards cycle, or until the standard is revised. Records concerning new, revised or 
reaffirmed continuous maintenance standards shall be retained for a minimum of five years or until the standard 
is completely revised or reaffirmed. Records concerning withdrawn standards shall be retained for at least five 
years from the date of withdrawal. 

7.5 CONSENSUS 

Evidence of consensus associated with the approval of an SCD by the PC shall be documented. 

7.6 CRITERIA FOR APPROVAL 

With respect to any proposal to approve, revise, or reaffirm an ASHRAE standard, evidence shall be considered 
that: 

(a) the applicable procedures were followed. 

(b) the SCD is within the scope of ASHRAE’s ANSI registered standards activities, 

(c) notice of the development process for the standard was provided to ANSI in accordance with PINS or 
its equivalent, 

(d) any identified conflict with another ASHRAE or American National Standard was addressed in 
accordance with the ANSI ER, 

(e) other known national standards were examined with regard to harmonization and duplication of content, 
and if duplication exists, there is a compelling need for the standard, 

(f) ANSI’s patent policy is met, 

(g) ANSI’s policy on commercial terms and conditions is met if applicable, 

(h) consensus was achieved, including evidence of the following: 

i. the applicable procedures were followed; 

ii. the SCD is within the scope of the registered standards activity; 

iii. declaration that conflicts with another ANS have been addressed per procedures; 

iv. a roster of the consensus body indicating the votes of each member, each member’s interest 
category and a summary of the vote; and 

v. identification of all unresolved negative views and objections, with the names of the objector (s), 
and a report of attempts toward resolution. 

(i) Any appeal meeting the criteria of B1 through B6 of Annex B was completed. 
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In addition, ASHRAE shall consider any evidence provided that the proposed standard is contrary to the public 
interest, contains unfair provisions, is unsuitable for national use, contradicts federal law(s), or is technically 
inadequate. 

ASHRAE shall not approve standards that duplicate existing or proposed Am erican National Standards unless 
there is a compelling need. 

7.7 CRITERIA FOR WITHDRAWAL OF STANDARD 

7.7.1 Requirements 

In considering a proposal for withdrawal of an existing ASHRAE Standard, the Standards Committee shall 
consider evidence that: 

(a) due process requirements were met, 

(b) consensus was achieved concerning the withdrawal of the existing standard, or consensus is lacking for 
its continued approval, 

(c) the proposal for withdrawal as an ANSI/ASHRAE Standard was provided to the administrator(s) of the 
appropriate USA Technical Advisory Group(s) and 

(d) any appeal to ASHRAE was completed. 

7.7.2 Withdrawal for Cause 

In the case of a proposal to withdraw an existing ASHRAE Standard for cause, the Standards Committee shall 
consider evidence that: 

(a) a significant conflict exists with an American National Standard, 

(b) ANSI’s patent policy was violated, 

(c) opportunity for consideration of revision was given but revision was not completed, or 

(d) the ASHRAE Standard: 

1. is contrary to the public interest, 

2. contains unfair provisions, 

3. is technically inadequate, or 

4. is unsuitable for national use. 

7.7.3 Other Bases for Withdrawal of Approval 

The ASHRAE Board of Directors or its designee also may withdraw approval of an ASHRAE SCD upon (a) 
advice of counsel, based on evidence of a legal nature, or (b) consideration of facts that have subsequently come 
to the attention of the Board. 

7.8 STANDARD PROJECT DISCONTINUANCE 

7.8.1 Project Discontinuation Due to Lack of Membership 

If a PC Chair and membership are not submitted by the TC or SPLS Liaison within twelve months after the 
project is approved, the MOS shall: 

a) automatically discontinue if this is a new project where the formation of a PC and TPS have been 
approved, or 

b) where a revision committee has been authorized, automatically refer the disposition to SRS for either 
reaffirmation publication public review or withdrawal public review. 

Waivers for project discontinuation shall be approved by SPLS and StdC. If the project is discontinued 
ASHRAE shall notify ANSI. 

7.8.2 Project Discontinuation Due to Lack of Performance 
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If the PC has not officially met for 12 months or is not advancing the development of the SCD in a timely 
ma nn er then the SPLS Liaison shall determine whether another Chair should be sought or, whether the matter 
should be sent back to PPIS to re-evaluate the need for the project. If the project is discontinued ASHRAE shall 
notify ANSI. 

7.9 Final Notice 

Notice of the final action on standards shall be announced on the ASHRAE web site. 

7.10 Emergency Interim Standards Action 

Emergency Interim Standards Action may be taken by the Society President, without completing all elements of 
due process, on an ASHRAE standard that has been published or has received publication approval by the Board 
of Directors. An Emergency Interim Standards Action has effect for limited duration and is for the exclusive 
purpose of correcting errors, other than errata, when failure to take timely corrective action would: 

a) substantively undermine the purpose or technical credibility of the standard, taken as a whole, or 

b) constitute undue risk to health or safety of the public or users of the standard. 

The Manager of Standards shall notify ANSI if Emergency Interim Standards Action has been taken on a 
published or candidate American National Standard. 

When an Emergency Interim Standards Action is taken, the Standards Committee shall initiate concurrent 
development of a revision or addendum, or initiate withdrawal procedures, to permanently correct the problem 
using ASHRAE’s consensus procedures. If corrective standards action is not approved by the Board of 
Directors for publication within two years, the Emergency Interim Standards Action shall be immediately 
terminated. (See Annex D .) 

7.11 Interpretation Requests of Standards 

Interpretation requests for a standard must be submitted to the MOS in writing. The Assistant Manager of 
Research & Technical Services or the Chair of the current or past cognizant PC or the Chairs designee may 
respond in writing to = written requests for unofficial personal interpretations. Cognizant SSPCs, if they exist, 
and SPCs that have not yet been disbanded will be asked to respond to requests for official interpretations in 
writing. If no PC exists, StdC will form an Interpretations Committee (IC) to respond. Procedures for 
interpretations of published SCDs are provided in StdC MOP Reference Manual Section 10. An issuance or 
revision of an official interpretation requires affirmative votes for the majority of the memberships of each 
approving and of at least two-thirds of those voting, excluding abstentions. 

7.-12 Interpretation Requests of ASHRAE Standards Development Procedures 

Interpretations requests for ASHRAE’s standards development procedures must be submitted to the MOS in 
writing. ASHRAE Staff may respond in writing to written requests for unofficial personal interpretations. 
Requests for official interpretations of procedures shall be submitted to PPIS. An issuance of an official 
interpretation requires affirmative votes for the majority of the memberships of PPIS and of at least two-thirds 
of those voting, excluding abstentions. 

8 PROCEDURES FOR SYNCHRONIZATION OF THE ASHRAE AND INTERNATIONAL 
STANDARDS REVIEW AND APPROVAL PROCESS 

When opportunities arise, the Standards Committee will encourage PCs to synchronize the review and approval 
process for ASHRAE and international standards consistent with ANSI procedures. If it is recommended that 
ASHRAE should use the expedited procedures for the identical adoption of an International Standards 
Organization (ISO) or International Electrotechnical Commission (IEC) standard the procedures in ANSI 
Procedures for the National Adoption of ISO and IEC Standards as American National Standards shall apply. 
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9 PATENTS 

ASHRAE agrees to comply with the Patent Policy as stated in ANSI Essential Requirements. 

10 COMMERCIAL TERMS AND CONDITIONS 

ASHRAE agrees to comply with the Commercial Terms and Conditions policy as stated in ANSI Essential 
Requirements. 

11 ANTITRUST POLICY 

ASHRAE agrees to comply with the Antitrust Policy as stated in ANSI Essential Requirements. 

12 PINS 

At the initiation of a project to develop or revise an ASHRAE Am erican National Standard, ASHRAE shall use 
the ANSI Project Initiation Notification System (PINS) form. Comments will be addressed in accordance with 
clause 2.5 of the current version of the ANSI ER. 
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This normative annex is part of the Procedures (PASA) 

ANNEX A: DEFINITIONS, ABBREVIATIONS AND ACRONYMS, AND CLASSIFICATIONS 


A1 DEFINITIONS 

addenda: revisions to a standard in the form of a supplement. 

alternate organizational representative (AOR): an individual empowered by an organizational member of a 
project committee to act on their behalf in the activities of the project committee when the representative of the 
organizational member is absent. 

annex: an appendix or attachment. See informative annex and normative annex 

balance: a condition existing when a) no single interest category constitutes more than one-third of the 
membership of a consensus body dealing with safety or b) no single interest category constitutes a majority of 
the membership of a consensus body. (Also see 7.3.3) 

clause: the basic component in the subdivision of the text of a standard. See subclause and 
paragraph. 

code intended standard: A standard intended to be adopted as a code using code language. 

code language document: A document that presents a set of requirements related to the design, 
application, or use of HVAC&R and related technologies where all or portions of the document may 
be enacted as mandatory enforceable requirements by a political jurisdiction. Portions intended to be 
enforced (normative) are written in mandatory, enforceable language. Portions not intended to be 
enforced are identified as informative and are to be located in informative notes, in informative 
annexes (appendices) or in other advisory documents. See annex, informative annex, informative 
notes and normative annex. 

cognizant TC/TG/TRG: the ASHRAE Technical Committee, Task Group, or Technical Resource Group 
within whose scope a particular standard’s technical content most logically falls. The cognizant TC/TG/TRG 
provides technical advice to the Standards Committee when a Standard Project Committee does not exist. 

conflict (between standards): refers to a situation where, viewed from the perspective of an implementer, the 
terms of one standard are inconsistent with the terms of another standard such that implementation of one 
standard necessarily would preclude proper implementation of the other standard in accordance with its terms. 

Conflict of interest: any incompatibility between an individual’s private interests and his or her fiduciary 
duties as an ASHRAE volunteer. 

consensus: substantial agreement, in the judgment of a duly appointed authority, reached by directly and 
materially affected interest categories. Substantial agreement means much more than a simple majority, but not 
necessarily unanimity. Consensus requires that all views and objections be considered, and that an effort be 
made toward their resolution. It is not required that each separate interest subcategory reach consensus on the 
standard. For ASHRAE standards projects and any jointly sponsored standards projects that use ASHRAE 
Procedures, the project committee is the consensus forming body. “Duly appointed authority” means the Board 
of Directors of ASHRAE and, in the case of jointly sponsored standards, the Boards of Directors of ASHRAE 
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and the joint sponsor(s). For American National Standards, “duly appointed authority” means the ANSI Board 
of Standards Review. 

continuous maintenance: maintenance of a standard by an SSPC for which procedures have been established 
to consider and process proposed changes as they are received. 

dominance: a position or exercise of dominant authority, leadership, or influence by reason of superior 
leverage, strength, or representation to the exclusion of fair and equitable consideration of other viewpoints. 

draft types: 

advisory public review draft: a draft submitted for public review that contains unusual, potentially 
controversial or new elements that the project committee believes would benefit from increased public 
scrutiny prior to finalizing the draft for publication public review. 

publication public review draft: a draft approved for public review that will proceed directly to publication 
if, as a consequence of the review, no substantive changes are made to the draft. 

working draft: an unapproved draft produced for consideration by the project committee or a subcommittee. 

due process: a course of proceedings carried out in accordance with established rules and principles. Due 
process allows for equity and fair play for all participants. It means that any person with a direct and material 
interest in a standard has a right to participate by (a) expressing a position and its basis, (b) having that position 
considered, and (c) appealing if adversely affected. 

Emergency Interim Standards Action: action taken by the Society President, without completing all elements 
of due process, on an ASHRAE standard that has been published or has received publication approval by the 
Board of Directors. An Emergency Interim Standards Action has effect for limited duration and is for the 
exclusive purpose of correcting errors, other than errata, when failure to take timely corrective action would: 

(a) substantively undermine the purpose or technical credibility of the standard taken as a whole, or 

(b) constitute undue risk to health or safety of the public or users of the standard. 

errata: a list of errors discovered after a document is published. 

Examples: typographical errors 
misprints 
misspellings 
grammatical errors 

omission of material approved by the StdC 
erroneous inclusion of material 

fast track: an approval procedure for a standards committee document that meets these criteria: 

a. there are no negative votes within the PC; 

b. no credible threat of legal action (in writing) against ASHRAE has been made related to the 
proposed draft; 

c. the proposed draft is not related to a Policy Level Standard (Policy Level PC Chair may request an 
exception. The SPLS Chair must grant or deny the exception within ten working days of 
submittal); and 

d. the SPLS Liaison has not notified the MOS within ten calendar days, from the receipt of the 
package, with specific justification, that the PC has violated due process. 

(See normal track) 


JA1799 



Case l:13-cv-01215-TSC Document 122-8 Filed 12/22/15 Page 121 of 174 
USCA Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 49 of 573 

five-year review: a review of need for standards action, scheduled so that processing and final approval of the 
resulting recommended action may reasonably be expected within five years from the date of Board approval of 
publication of ASHRAE Standards and Guidelines, or within five years of ANSI approval as an American 
National Standard. 

foreword: introductory remarks, not part of the standard. 

independent substantive change (ISC): a substantive change that is independent of any other substantive 
change and that does not significantly affect any other requirement in the standard. See substantive change. 

informative annex: additional information of a non-mandatory nature. Changes to informative annexes are 
considered non-substantive. Informative annexes can be changed or deleted without requiring public review. See 

normative annex and notes. 

informative language: language used in those elements of an SCD for which compliance is not required, often 
characterized by the use of “should” or “may.” 

Informative notes: explanatory information, appearing in a standard, that does not contain requirements or any 
information considered indispensable for the use of the standard. I nf ormative notes are to begin with the word 
“(Informative Note(s))” and be placed after the section of the standard to which the note applies. If the 
“informative note” is more than two sentences, the information shall be placed in an in form a tive a nn ex and 
referred to by the informative note. Where there is more than one informative note, the notes must be numbered 
sequentially. 

interest: the perspective of a member of a project committee, as judged by his or her present and past sources of 
income, fees, or reimbursements of related expenses, in the context of the purpose and scope of the project 
committee. The perspective may also be judged by the recorded views of the individual, or of any organization 
he/she is employed by or of which he/she is a member. 

interest category: a category identified to represent a specific interest. 

interest categories: a classification of project committee member interests. For some projects, it may be 
appropriate to designate subcategories of one or more interest category. Default interest categories are: 

Producer: A member who represents the interest of those that produce materials, products, systems, or services 
covered in the project scope. 

User: A member who represents the interest of those that purchase or use materials, products, systems, or 
services other than for household use covered in the project scope. 

General: A member who ca nn ot be categorized in any other approved interest category covered in the project 
scope. 

Additional examples of interest categories and subcategories that have been used can be obtained from the 
MOS. 

International Organization for Standardization (ISO): an international non-treaty standards organization 
based in Geneva, Switzerland. Its members, national standards bodies, promulgate standards covering all fields 
except electrical. The American National Standards Institute is the U.S. member body. 
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international organizational liaison (IOL): a non-voting representative of an international trade or 
professional organization, international standards committee, or other group with an interest in the work of the 
PC. 

interpretation: the written explanation of the meaning of specific provisions of a standard or guideline, as 
determined by the project committee or the interpretations committee in response to an inquiry. 

interpretations committee (IC): a committee of technically qualified individuals whose function is to interpret 
an ASHRAE standard or guideline. 

mandatory language: language that prescribes the requirements of a standard in a manner that is clear and 
unambiguous. It provides a basis for determining, without a doubt, whether or not compliance with the standard 
has been achieved. It is often characterized by the use of “shall” or “must.” 

non-substantive changes: non-substantive changes are limited to: 

a) changes to the main body of text of the standard or guideline to update inform a tion references; to 
correct errata, punctuation or grammar, typographical errors or style; or to add equivalent SI or I-P 
values; 

b) changes to the foreword, membership rosters, or other adjuncts not part of the standard or guideline; 
and 

c) changes to in formative appendices or a nn exes not part of the standard or guideline. 

normal track: an approval procedure applied to a standards committee document that meets one or more of 
these criteria: 

a) receives one or more negative votes upon approval for publication or 

b) where ASHRAE receives a written legal threat or 

c) is a policy level standard. 

(See fast track) 

normative annex: additional information of a mandatory nature which, for reasons of convenience, is placed 
after the main body of the document. See informative annex. 

Non-Voting Member (NVM): An NVM is an additional type of membership for PCs not formally organized 
into subcommittees. NVMs are not eligible to vote on PC motions. NVMs are not included in interest balance 
or quorum requirements. 

organization: a group of people representing a particular interest such as a trade association, public interest 
group, or government agency. 

Organizational Member (OM): An OM is an organization with a voting representative on the PC that 
represents the interests of that particular organization rather than serving as an individual. 

policy level document: a standards committee document designated as “policy level” by the Board of 
Directors or the Board’s designee. 

Project Committee Voting Member (PCVM): PCVMs are eligible to vote on PC motions. PCVMs are also 
eligible to vote on subcommittee motions to which the PCVM is appointed. 

Project Subcommittee Voting Member (PSVM): PSVMs are eligible to vote on subcommittee motions to 
which the PSVM is appointed. PSVMs are not eligible to vote on PC motions. PSVMs are not included in 
interest balance and quorum requirements for the PC. 



PASA - Procedures for ASHRAE Standards Actions 
ANSI Approved: April 29,2015 


JA1801 



Case l:13-cv-01215-TSC Document 122-8 Filed 12/22/15 Page 123 of 174 
USCA Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 51 of 573 

periodic maintenance: review and action on a no mi nal 5-year cycle to revise a standard or to reaffirm or 
withdraw a standard. 

project committee (PC): a Standard Project Committee or Standing Standard Project Committee. 

public review comment: views and/or objections to standards or addenda to standards submitted in accordance 
with procedures specified in the public review draft during a public review. 

rating: the assigned values of those performance characteristics, under stated conditions, by which a piece of 
equipment may be chosen to fit its application. These values apply to all equipment of like no mi nal size and 
type (identification) produced by the same manufacturer. 

standard rating: a rating based on tests performed at standard rating conditions. 

application rating: a rating based on tests performed at application rating conditions (other than standard rating 
conditions). 

rating conditions: a set of operating conditions under which a level of performance is determined or 
measured. 

standard rating conditions: rating conditions used as the basis of comparison of performance characteristics, 
shall: a verb use to indicate a requirement, 
should: a verb used to indicate a recommendation. 

SPLS liaison: a member of the Standards Project Liaison Subcommittee (SPLS) assigned to act as a 
Standards Committee advisor to a project committee. 

standard: a document established by authority or rule that defines properties, processes, dimensions, materials, 
relationships, procedures, concepts, nomenclature, or test methods for rating purposes. Adherence to due 
process in its development and achievement of consensus are conditions of approval. 

standards action: an action recommending or approving publication of a new, revised, or reaffirmed standard 
or withdrawal of a standard. 

Standards Action: a periodical published by ANSI to inform interested persons about American National 
Standards (ANSs), including proposals to initiate projects to develop or revise ANSs, announce intent to 
reaffirm or withdraw existing ANSs, communicate status of international standards, a nn ounce public review of 
proposed or revised procedures of ANSI accredited standards developers, etc. 

Standard Project Committee (SPC): a committee of technically qualified individuals with a balanced 
representation of interests whose function is to formulate, review, reaffirm, or revise an ASHRAE standard. The 
SPC is the consensus-forming body and is responsible for the technical content of the standard. It is discharged 
upon publication of the standard. 

Standing Standard Project Committee (SSPC): a committee similar in membership and function to a 
Standard Project Committee except that the committee has a continuing assignment of duties and responsibilities 
with respect to a standard. It is expected to provide addenda as needed, generate revision on a regular basis, and 
render interpretations. 
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subcommittee, project committee: a group of individuals appointed by the project committee chair from 
among the project committee membership who vote on subcommittee activities and whose responsibility it is to 
develop drafts of one or more assigned sections of a standard, annexes, or addenda; develop draft responses to 
requests for interpretation; or develop proposed responses to comments resulting from public review; all 
submitted as recommendations for action by the parent project committee. 

substantive change: a change that involves an important (has value, weight or consequence), fundamental (is 
the foundation, without which it would collapse), or essential (belongs to the very nature of a thing) part or 
changes the meaning of the material or that directly and materially affects the use of the standard. Changes that 
may be found substantive when examined in context. 

(a) “shall” to “should” or “should” to “shall;” 

(b) addition, deletion or revision of mandatory requirements, regardless of the number of changes; or 

(c) addition of mandatory compliance with referenced standards. 

Changes or deletions made to portions of a draft not intended as part of the approved standard (e.g., a foreword, 
informative a nn ex or note), are not considered substantive. 

See independent substantive change. 

system of units: inch-pound units (I-P) or International System of Units (SI). 

Technical Resource Group (TRG): a committee of technical experts appointed by TAC, to prepare or review 
technical material for standards, the ASHRAE Handbook, Journal articles and technical papers. 

unit conversions - definitions: 

alternate system of units: the system of units listed second (expressed in parentheses when dual systems, I-P 
and SI are used, expressed in either consistent rational or equivalent values.) 

equivalent: exact arithmetic conversions, also called “soft conversion.” 

primary system of units: the system of units listed first (expressed in rational values). 

rational: based on, or derived from, logical or coherent numbers. Rational values are usually, but not 
necessarily, rounded numbers. Rational values are not necessarily bound by mathematical equivalency of the 
primary and secondary units systems. The conversion process is sometimes called “hard conversion.” 

unresolved public review commenter: an individual who, during the comment period, submitted public review 
comments to a proposed or revised draft standard, guideline or addendum, was not satisfied with the committee 
response to those comments and, within the time period and procedure specified in the response, requested to 
remain “unresolved”. 
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A2 ABBREVIATIONS AND ACRONYMS 


ANS 

American National Standard 

ANSI 

American National Standards Institute 

ASHRAE 

American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. 

BOD 

Board of Directors 

CIS 

Code Interaction Subcommittee 

IC 

interpretations committee 

IOL 

international organizational liaison 

I-P 

inch-pound units: units using inches, pounds, and other designations; as opposed to SI 
units in the metric system. Examples are: foot, Btu, horsepower, gallon. 

ISC 

independent substantive change 

ISO 

International Organization for Standardization 

MOS 

Manager of Standards 

PC 

Project committee. Refers to both an SPC and an SSPC. The use of this acronym means that a 
procedure applies to both. 

PCVM 

project committee voting member 

PPIS 

Planning, Policy and Interpretation Subcommittee 

PSVM 

project subcommittee voting member 

SCD 

Standards Committee Document 

SI 

Le Systeme International d’Unites; the international agreement on the metric system of units. A 
practical system of units divided into three classes: base units, derived units and supplementary 
units. The base units are composed of the units of the following seven quantities: length (meter), 
mass (kilogram), time (second), electric current (ampere), thermodynamic temperature (Kelvin), 
amount of substance (mole), and luminous intensity (candela). 

The second class of SI units contains derived units, i.e., units that can be formed by combining 
base units according to the algebraic relations linking the corresponding quantities. The names and 
symbols of some units thus formed in terms of base units can be replaced by special names and 
symbols which can themselves be used to form expressions and symbols of other derived units. 

A third class of SI units, called supplementary units, contain the SI units of plane and solid angle. 
(Ref. Le Systeme International d’Unites) 

Wm 

23 
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SPC 

Standard Project Committee. The use of this acronym means that a procedure applies only to an 
SPC and not to an SSPC. 

SSPC 

Standing Standard Project Committee. The use of this acronym means that 
only to an SSPC and not to an SPC 

a procedure applies 

SPLS 

Standards Project Liaison Subcommittee 


SRS 

Standards Reaffirmation Subcommittee 


StdC 

Standards Committee 


TAC 

Technical Activities Committee 


TC 

Technical Committee appointed by the TAC 


TRG 

Technical Resource Group appointed by TAC 


TG 

Task Group appointed by the Technical Activities Committee 


TPS 

Title, Purpose and Scope 
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This normative annex is part of the Procedures (PASA) 


ANNEX B: APPEALS OF BOARD OF DIRECTORS’ STANDARDS ACTIONS OR INACTIONS 
B1 SCOPE 

This procedure applies to appeals of ASHRAE Standards and of jointly sponsored standards for which 
ASHRAE is the lead sponsor. 

B2 APPEALABLE MATTERS 

An action or inaction of the Board of Directors (BOD) to adopt a new ASHRAE standard, an addendum 
to an existing standard, or to revise, reaffirm, or withdraw an existing ASHRAE standard is subject to 
appeal. 

B3 WHO MAY APPEAL 


Any person directly and materially affected by the publication of a new, revision, reaffirmation, or 
withdrawal of an ASHRAE standard, or lack of such action, may appeal the BOD action or inaction. The 
appellant must be an unresolved public review commenter, associated with a new, revision, reaffirmation 
or withdrawal of the ASHRAE standard being appealed, or a PC member who cast a negative vote with 
reason(s) in relation to hi s /her vote on the consensus body associated with the creation, revision, 
reaffirmation or withdrawal of the ASHRAE standard being appealed. 

B4 SCOPE OF APPEAL AND BURDEN OF PROOF 

An appeal of a BOD standards action or inaction shall be solely based upon procedural grounds. When 
appeals are filed, the appellant shall demonstrate that ASHRAE Standards development procedures were 
not followed. Appeals arguments that are based on actions that took place in previous revision cycles will 
not be considered. 

B5 CONTENT OF APPEALS 

Each appeal shall: 

(a) Identify the appellant, and include the appellant’s contact information; 

(b) Substantiate that the appellant is directly and materially affected by action(s) being appealed; 

(c) Identify with precision the standard or portions thereof, and the procedure(s), alleged improper 
action or inaction appealed; 

(d) State concisely the basis for the appeal, the remedial action requested, and the nature of any injury 
to appellant which might accrue from the matter appealed; 

(e) Include any summary supporting data or documentation relied upon as the basis for the appeal; 

(f) Consolidate information to be as concise as possible; 

(g) Only include information that was made available to the PC prior to the final vote of the PC; 

(h) Include the filing fee. 
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B5.1 FILING FEE 

Each appeal shall be accompanied by a filing fee in the amount established by the Technology Council. 
The filing fee is predetermined and shall be listed on the Appeals Submittal Form. The fee may be waived 
or reduced by the Chair of the Technology Council upon sufficient evidence of hardship submitted by the 
appellant. If the filing fee is not submitted by the appeal filing deadline date by the appellant then the 
appeal shall be dismissed unless an exception has been granted prior to the close of business on the filing 
deadline date. 

B5.2 COPIES 

It shall be the responsibility of the appellant to submit an electronic copy and if requested by the Manager 
of Standards, up to twenty-five (25) paper copies of each appeal filed at the time of the original electronic 
submittal. 

B6 NOTIFICATION PROCEDURES 


Within 15 days following BOD action on a standard, that results in approval of a new, revision, 
reaffirmation or withdrawal of a standard or addenda to a standard, the Manager of Standards (MOS) 
shall notify in writing (including electronic communication) all unresolved public review commenters 
and/or a PC member who cast negative votes with reason(s) in relation to his/her vote on the consensus 
body of the BOD action and inform them of their right to appeal that action. 

B6.1 An appeal, must be received by the Manager of Standards (MOS) of ASHRAE within 15 
working days of the date on the notification letter regarding the BOD action. The Chair of the Appeals 
Board may grant an extension, if requested prior to the close of the initial 15 working day period and if 
sufficient justification is provided. 

B6.2 Normally, any standards action by the BOD will be suspended during pendency of appeal(s), 
appropriately filed. The President may, however, maintain the BOD action until and if the Appeals Panel 
decides to dismiss the appeal, without a hearing, up to a maximum of 90 days. If the Panel decides to 
dismiss the appeal without a hearing, the President may maintain the action until the next meeting of the 
Board of Directors. The appealed BOD action shall be immediately suspended if the Appeals Panel does 
not dismiss the appeal. 

B6.3 The MOS shall acknowledge receipt of the appeal, copy acknowledgement to the Chief Staff 
Officer, notify the President, and send copies of the appeal to the Appeals Board Chair and to the Chairs 
of Technology Council, Standards Committee and the Project Committee (PC) which developed or 
revised the standard, if applicable. Upon receipt of the appeal, an Appeals Panel will be established in 
accordance with Section B8 for the purpose of determining if the appeal will be heard or if the appeal will 
be dismissed without a hearing. 

B7 APPEALS BOARD 


B7.1 An Appeals Board and a chair of the Board shall be appointed by the ASHRAE President, with the 
approval of the Board of Directors. The Appeals Board shall have 15 members. The Appeals Board shall 
consist of past members of the BOD, past members of the Standards Committee or Technology Council, 
and/or persons who are knowledgeable about the ANSI Standards development process. 
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B7.2 Terms of Membership 

Terms shall be staggered so that approximately one-third of the membership of the Appeals Board is 
appointed each year. Members shall be appointed for a term of three years commencing on July 1, and 
shall be eligible for reappointment for one additional 3-year term, for a total of two consecutive terms. A 
member of the Appeals Board may serve beyond the normal two-term limitation if the member is serving 
as chair, provided the term of chair is contiguous with the six-year tenure as a member. The total 
maximum length of service under such circumstances would be nine years. 

B7.3 Vacancies 

A vacancy in the membership of the Appeals Board shall be filled for the remainder of the term by an 
individual appointed by the ASHRAE President. 

B7.4 Conflict of interest 

A member of the ASHRAE Appeals Board shall act at all times in a manner that promotes confidence in 
the integrity and impartiality of ASHRAE’s processes and procedures and should avoid a conflict of 
interest or the appearance of a conflict of interest in connection with all ASHRAE Appeals activities. 
Should the Appeals Board Chair have a conflict of interest with any appeal he/she shall select another 
member of the Appeals Board to serve is his/her place with respect to consideration of that appeal. 

If a materially affected party (either the appellant or the respondent) asserts that it believes a member of 
the ASHRAE Appeals Board has a conflict of interest, that materially affected party is required to state 
the reason(s) for its belief. That information shall then be forwarded to the member of the ASHRAE 
Appeals Board identified as having a possible conflict for that person’s response. If that member 
disagrees with the assertion, then the Chair of the ASHRAE Appeals Board shall make a final 
determination as to whether a conflict of interest exists. 

Members of the ASHRAE Appeals Board who are disqualified from a particular discussion shall not 
participate in the arguments, deliberations or decisions. 

B7. 5 When appeals of jointly sponsored standards are being considered by ASHRAE as lead sponsor 
or by ANSI, the joint sponsor shall assist in preparing or responding to appeals in its field of expertise. 


B8 CONSIDERATION OF APPEALS 

B8.1 When an appeal is received by ASHRAE Headquarters in accordance with Section B6.3 six 
members of Appeals Board shall be randomly selected from a pool of all Appeals Board members that do 
not have a conflict to hear the appeal. At least four of those selected shall be appointed as the Appeals 
Panel and the other 2 shall be appointed as alternates. The Appeals Panel alternates will participate in the 
hearing activities in the event that one of the four other members are unable to serve. The Appeals Board 
chair will chair the Appeals Panel. 

B8.2 Members of the Appeals Panel shall not have been a PCVM or PSVM on the project committee that 
is the subject of the appeal during the three years prior to the standards action under appeal. Members of 
the Appeals Panel shall not have voted on the draft that is the subject of the appeal as a member of the 
Standards Committee or Board of Directors. 

B8.3 The Appeals Panel shall first decide if the appeal shall be dismissed without a hearing. Non- 
compliance with Section B5 or lack of grounds for an appeal may be reasons for dismissal. To assist in 


:eilures for ASHRAE Standards Actions 
ved: April 29,2015 


27 


JA1808 



Case l:13-cv-01215-TSC Document 122-8 Filed 12/22/15 Page 130 of 174 
USCA Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 58 of 573 


this decision, the Appeals Panel Chair may request a rebuttal statement from the respondent (the Chair of 
the Standards Committee or his/her designee, or the Chair of the PC or his/her designee), as appropriate. 
The Appeals Panel Chair shall inform the appellant within 30 days of the receipt of the rebuttal whether 
the appeal will be dismissed without a hearing, decided after a hearing, or decided without a hearing. 

B8.4 If the appeal is not dismissed, the BOD action which has been appealed shall be immediately 
suspended, if not already suspended according to the first sentence of B6.2, and each claim in the appeal 
shall be considered separately and basic grounds given for each decision. The Appeals Panel shall decide 
whether a hearing is warranted or if a decision can be made and reported to the President on the appeal 
without a hearing. 

B9 HEARING OF APPEALS 
B9.1 Notice 

If the appeal is to be heard, the Appeals Panel chair shall arrange for consideration of the appeal by 
meeting, or documented telephone conversations. Both the appellant and the respondents (the Chair of the 
Standards Committee or his/her designee, or the Chair of the PC or the Chair’s designee, as appropriate) 
shall be given at least 45 days notice of the hearing date (from the date on the notification letter), location, 
and time for a hearing or 30 days notice of the hearing date (from the date on the notification letter) for a 
hearing conducted by conference call. The 30 or 45 days may be waived if the appellant and the 
respondents agree in writing (including electronic communication). During this period a rebuttal of the 
written statement of appeal shall be submitted to the MOS who shall distribute it to the Appeals Panel and 
to the Appellant. The rebuttal, if not previously requested, from the respondent(s) shall be due within 15 
working days of the date on the letter of notification. The Chair of the Appeals Panel may grant an 
extension if requested prior to the close of the initial 15 working day period and if sufficient justification 
is provided. The rebuttal statement shall be sent to the MOS, who shall distribute it to the appellant and 
the Appeals Panel. 

B9.2 The Hearing 

At the hearing, the appellant and respondent(s) shall provide the Chair of the Appeals Panel with 15 
copies of an outline of their oral presentation or a copy of what will be displayed for their electronic 
presentation. No new issues outside of those issues raised in the submitted appeal may be presented at the 
hearing. Only documentation that the Appellant/Respondent has already been given, which supports 
raised issues, will be permitted in the presentation. Both the Appellant and the Respondent are permitted 
to have people speak on their behalf (i.e.: experts). However, each party is only allowed a designated 
amount of time and that time will be shared by any and all people speaking for that party. No additional 
time will be granted for guests, speakers, experts, etc. 

B9.3 A Standards Committee Liaison and the BOD Ex-Officio member of the Standards Committee 
shall be invited by MOS to attend the hearing. The hearing shall be open to representatives of directly 
and materially affected persons, although the number of any interest group may be limited at the 
discretion of the Appeals Panel Chair. Anyone planning to attend the hearing shall notify the MOS within 
a minimum of 15 days prior to the hearing date. The deliberations of the Appeals Panel shall be held in 
Executive Session. 

B10 APPEALS PANEL DECISION 


The Appeals Panel shall decide within 45 days of the hearing, by majority vote, that the appeal, or any 
parts of the appeal, be upheld or denied. The Appeals Panel Chair shall, within 14 days following the 
Appeals Panel’s decision, notify the appellant(s), Chief Staff Officer, Director of Technology, Manager of 
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Standards, President, Chair of Technology Council, Chair of the Standards Committee, and Chair of the 
PC of the decision. The decision of the Appeals Panel to uphold, deny, or dismiss an appeal shall be 
final. If the appeal is dismissed or denied by the Appeals Panel, the action of the BOD, which was 
appealed shall become effective immediately. 
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This normative annex is part of the Procedures (PASA) 


ANNEX C: COMPLAINTS OF ACTIONS OR INACTIONS BY THE STDC, ITS 
SUBCOMMITTEES OR PCs 

In addition to formal appeal of BOD Standards actions or inactions (PASA Annex B), failure of the StdC, 
its subcommittee(s), or a PC to consider a written request may be addressed by writing to the MOS at any 
time. 

a) A written complaint shall be sent to the MOS and the MOS shall forward it to the Chair of the 
Committee in question. The MOS shall acknowledge receipt of the complaint (i.e., Subject 
Committee Chair). 

b) The Subject Committee Chair shall provide a written response to the complainant, with a copy to 
the MOS within 15 working days of receipt of the complaint. A waiver to the response period 
may be requested by the Chair or ASHRAE Staff to the Chair of the next higher body. (e.g. StdC 
Chair for a PC Chair). The waiver request shall be promptly addressed. 

c) The complainant shall notify the Subject Committee Chair and MOS in writing within 15 days 
from the receipt of the response whether or not the response resolves the complaint. If no 
response is received then the higher body, the complainant and the Subject Committee Chair will 
be notified that the complaint is resolved. 

d) If the response does not resolve the complaint, the complaint shall be forwarded to the next 
higher body. The next higher body shall place it on its next agenda for consideration but a 
meeting shall be called no later than 15 working days after receipt of the complaint. 

e) When the complaint has been heard by the next higher body, the Chair of that body shall notify 
the complainant in writing, with a copy to MOS, and to the Chair of the committee in question of 
the committee’s decision within 15 days. (The next higher body is the committee, which 
approves the actions of the committee in question). 

f) The final level to resolve the complaint shall conclude at Technology Council. Should the 
unresolved complaint reach Technology Council, Technology Council shall have the authority to 
decline to hear the complaint. 
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This normative annex is part of the Procedures (PASA) 


ANNEX D: UNITS POLICY 

The units use or application policy shall include, as a minimum, time-dated directions on the use of SI 
and I-P in all ASHRAE publications. 

TC 1.6 shall serve as the authority on SI and I-P usage and application. 

Research projects; codes, standards, guidelines, and addenda thereto; special publications; Insights 
articles; Journal articles; and Handbooks shall be prepared using the International System of Units (SI) 
and/or inch pound units (I-P) in formats approved by the Publishing and Education Council. 

The Publishing and Education Council shall review annually the approved formats to be used in 
AHSRAE publications, considering suggestions from members and committees, and shall establish any 
changes in the approved formats. 

The Publishing and Education Council shall consider this Units Policy annually and shall recommend to 
the Board of Directors the formats to use in ASHRAE publications. 

(a) The format for ASHRAE publications shall be dual units, except in cases determined by the 
Publishing and Education Council, where two separate versions are to be published, where one is 
rational SI and the other is rational I-P. For selected ASHRAE standards and guidelines, the 
Standards Committee may approve use of SI units only. 

(b) In dual unit publications, the units used in calculating the work being reported shall be listed first. 
The alternate system of units should follow in parentheses. Authors shall round off equivalents in 
the alternate system of units so that they imply the same accuracy as is implied with primary 
units. Exceptions require the approval of the Director of Publishing and Education. Handbook 
volumes shall be published in separate SI and I-P editions. 
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This normative annex is part of the Procedures (PASA) 

ANNEX E: Procedures - Emergency Interim Standards Action 
El Justification 

The burden of demonstrating need for an Emergency Interim Standards Action rests with the proposer. 
Interested persons may submit proposals for Emergency Interim Standards Actions to the MOS. 

Proposals must include the following information: 

a) identify the proposer, affiliation and contact information: 

b) identify the standard or guideline and clause containing the error, 

c) describe the error claimed and provide supporting information or data, if any, 

d) recommend a change in text, equation, etc. that would eliminate the error or reduce it to 
acceptable limits and provide supporting information or data, if any, 

e) show compliance with the criteria of Section 6.9(a) or 6.9 (b), and 

f) identify the type of harm that has been or may be caused by the error. 

Proposals that meet the criteria of Section 6.9 shall be forwarded to the body designated in E5. 

E2 PC or PPIS Recommendation 

When a PC having jurisdiction exists, the PC shall submit a recommendation to the MOS on disposition of a 
proposed Emergency Interim Standards Action at a PC meeting or by letter ballot within 14 days. When a PC 
does not exist, PPIS shall act in lieu of a PC. 

E3 MOS Recommendation 

If the PC or PPIS fails to submit a recommendation within 14 days, the MOS shall submit his/her 
recommendation. 

E4 Review and Comment 

Upon receipt of a recommendation resulting from E2 or E3, the MOS shall circulate the proposed 
Emergency Interim Standards Action and recommendation within seven days to the StdC, the Director of 
Technology, and the MOS for review and comment. 

E5 President Will Act 

A package composed of the proposed Emergency Interim Standards Action, recommendations resulting 
from E2 or E3, and recommendations from the Standards Committee Chair, Director of Technology, and 
MOS, whether positive or negative, shall be submitted within 14 days of receipt by the MOS for the 
President’s consideration and decision. 

E6 Notifications 

The MOS shall issue notification of the President’s decision to the proposer, the Editor of the ASHRAE 
Journal, and ANSI, and shall in itiate implementation of the decision as appropriate. 
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Under the ANSI Organization Method (PASA), dated June 30, 1994 
superseded all previous documentation for communicating ASHRAE’s 
procedures as a basis for continuation (re-accreditation) under the ANSI 
Organization Accreditation Method. 

June 29,1994 

A 

The first revision was approved by the Board of Directors on February 2, 
1995 and incorporated nine changes for clarifications and in response to 
comments resulting from ANSI public review of PASA. 

February 2,1995 

B 

On April 28,1995, staff incorporated clarifying revisions to the figures in 
informative Appendix C and added a new Figure 6. ANSI reaccredited 
ASHRAE on August 4,1995 based on this edition. 

April 28,1995 

C 

The third revision was approved by the ASHRAE Board of Directors on 
June 27, 1996 and incorporated twelve changes in response to 
recommendations in the ExSC Appeals Panel decision letter dated April 

23, 1996, the draft ANSI Report of Audit of ASHRAE procedures and 
operations dated June 10,1996, and the need for clarification. 

June 27,1996 

D 

The fourth revision included broadening the section on membership, by 
allowing for possibilities for organizational membership. Additionally, 
this revision incorporates some changes involving written responses to 
commenters and resolution of commenters. The ASHRAE Board of 
Directors approved this version January 27,1999. ANSI reaccredited 
ASHRAE on May 7,1999. 

January 27,1999 

E 

This revision includes changes to allow the newly approved Board Policy 
Committee for Standards to have oversight authority for certain project 
committees. It also deleted references to specific sections of ANSI 
procedures so that revision to PASA would not be necessary when section 
numbering in the ANSI procedures changed. The ASHRAE Standards ftp 
site (fto.ashrae.ore/stds-info) is now utilized as the means for advertising 
standards activities, in lieu of the ASHRAE Journal. The records 
retention policy has been clarified, and the references to formal Mediation 
Meetings have been removed. Finally, the appeals procedures were 
modified to more closely match the ANSI appeals procedures. 

ANSI reaccredited ASHRAE on November 21, 2001. 

June 29, 2000 

F 

Editorial revision of Section 6.2.1.2 made to reflect the oversight authority 
of the Board Policy Committee for Standards. 

February 1,2001 

G 

Editorial revision of Sections 4.1 and 6.2 made to reflect removal of 
Appendix C. 

June 27, 2001 

H 

This revision included changes in Sections 6.2.1.2-6.2.1.3.2 to require 
letter ballot votes for publication approval by the Consensus Body. 
Appendix B3 was revised to further clarify the appeals process. 

January 17,2002 


This revision includes changes in Sections 5, 6.3.6, 6.2.1.3, the Appendix 

A1 definition of “balance," and the addition of Section 8 - Patents. 
Appendix B was revised to assign final approval of appeals to the Board 
Policy Committee for Standards. 

June 27, 2002 

J 

This revision includes changes in sections 4.1, 6.2.1, 6.2.2, Al, and A2 to 
change the reference from Technical Evaluation Committees (TEC’s) to 
Technical Resource Groups (TRG’s). Changes were also made to sections 
6.3.1.3 and 6.7 to remove the reference to the ASHRAE ftp site. 

January 30, 2003 
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K 

This revision includes the addition of a sentence to section 6.3.4.2 
(Complaints of Inactions) that clarifies who addresses complaints. 

July 3, 2003 

L 

This revision addresses the following issues: the clarification of ANSI 
requirements, removal of the Board Policy Committee for Standards 
(BPCS) oversight responsibility and changes to the appeals process. 

July 1, 2004 

M 

This revision replaces language that was inadvertently deleted in the 
Nashville revision, to provide the provision to appoint the Appeals Panel 
Chair. 

June 30,2005 

N 

This revision includes revisions to section 4.3.2 (Joint Sponsorship) so that 
the MOS can negotiate terms of the joint sponsorship agreements. 

Changes were made to B6, B9.1, and B9.2 to clarify the appeals process. 
Section B9.2, The Hearing, was added to clarify the rules during the 

Appeals hearing. 

January 26, 2006 

O 

This revision includes revisions to Section 4.3.2 (Joint Sponsorship) and 
removes approval by Technology Council and the BOD of the final 
negotiated cosponsorship agreement. 

March 20, 2006 

P 

This revision includes adding the terms “including electronic 
communication” to Section 4.2.1.1, Section 6.4.3.2, and B9.1. This also 
includes revisions to 4.3.2 to clarify the language regarding Joint 
Sponsorship approval. Section 9, Commercial Terms and Conditions, was 
added. The definition of contact information was added to Appendix A. 
Revisions were made to Section B5 to add request for contact information 
and to limit the materials that are allowed in appeals. 

June 29,2006 

Q 

This revision includes adding the cm records retention policy to Section 
6.3.8, adding Section 6.7, Interpretation Requests, and adding Annex C, 
Units Policy to PASA per the request of ANSI. 

March 2, 2007 

R 

This revision in the Introduction section includes, moving part of the 
information to an informative forward. 

October 24,2008 

S 

This revision in Section 3, changes Appendix to Annex. 

October 24, 2008 

T 

This revision in Section 5 deletes text from ANSI Essential Requirements 

October 24, 2008 

U 

This revision in Section 6.2.1, (Approval) includes Technology Council in 
the approval of publication drafts 

October 24, 2008 

y 

This revision includes in Section 6.2.1.2, (Voting Requirements for 
Standards Actions), changing the vote from letter ballot to recorded votes, 
adding Technology Council and allowing the Board or its designees to 
vote. 

October 24, 2008 

w 

This revision includes the deletion of Section 6.2.1.3 

October 24, 2008 

X 

This revision includes in Section 6.2.2 (Modification of Standards) the 
addition of the need for a revision to a standard. 

October 24, 2008 

Y 

This revision to Section 6.2.4 (Substantive Changes) deletes the entire 
section. 

October 24, 2008 

z 

This revision to Section 6.3.2 (Balance and Lack of Dominance) changes it 
to read like ANSI Essential Requirements 2008. 

October 24, 2008 

A 

This revision to Section 6.3.3 (Interest Categories) deletes language in 
order to simplify the interest categories. 

October 24, 2008 

AB 

This revision to section 6.3.4.1 (Appeals to BOD), includes the change 
from Appendix to Annex and includes the deletion of identifiable, realistic 
and readily available text. 

October 24, 2008 
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AC 

This revision to Section 6.3.6(Consideration of Comments Received) 
includes the addition of language specifying Public Review. Title reflects 
as Consideration of Public Review Comments Received and within the 
paragraph, “public review” was inserted. 

October 24, 2008 

AD 

This revision to Section 6.4 (Consensus) was rewritten to require 
documentation that the consensus is in accordance with ANSI Essential 
Requirements and PASA. 

October 24, 2008 

AE 

This revision to Section 6.5 (Criteria for Approval) modified letter (i) to 
change Appendix to Annex. 

October 24, 2008 

AF 

This revision to Section 8 (Patents) was editorially modified. Removed the 
text “such” and “or guideline” from the first sentence. 

October 24, 2008 

AG 

This revision to Appendix A includes: 

• The deletion of ASHRAE Information Representative 

• Modification of the definition of balance by deleting “dealing with 
product standards.” 

• Modified definitions of continuous maintenance definition and 
interest category 

• Modified definition of informative annex 

• Modified interest categories definition; deleted the definition for 
all subcategories, user, producer and general 

• Added a Method of Test Standard definition 

• Modified the definition for normative annex 

• Modified the definition for public review comment 

• Deleted testing standard definition 

• Modified unresolved commenter definition 

• Deleted Section A3 

October 24,2008 

AG 

This revision to Appendix B includes: 

• Appendix B2, deleted the availability of EISA’s to be appealed to 
the Board as this can be handled through the complaint process 

• Appendix B3, modified who the appellant must be and how the 
vote should be casted 

• Appendix B5.2, inserted the word “copies” 

• Appendix 6, specified who the MOS should notify, public review 
commenters and/or a PC member who cast negative votes with 
reason(s) in relation to his/her vote on the consensus body 

• Appendix B10, added language requiring that the Appeals Panel 
vote within 45 days of the hearing whether or not the appeal is 
upheld or denied. 

October 24, 2008 

AH 

This revision to Section 4.1 includes the addition of the word publishing. 

February 25,2011 

AI 

This revision to Section 6 title includes the addition of language for 
discontinuing ASHRAE standards. 

February 25, 2011 

AJ 

This revision to Section 6.3.6 includes language in the first paragraph 
straight from the PC MOP regarding information about the online 
comment database. 

February 25, 2011 
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AK 

The revision to Section 6.5 includes added and deleted language. The 
additions are from the ANSI Essential Requirements and are listed below: 

• Notice of the development process for the standard was provided 
to ANSI in accordance with PINS or its equivalent 

• Identification of all unresolved negative views and objections, 
with names of the objector(s), and a report of attempts toward 
resolution 

• The standard is within the purpose and scope approved by the 
Standards Committee 

• ... .and if duplication exists, there is a compelling need for the 
standard 

• ANSI’s policy on commercial terms and conditions is met if 
applicable 

The deletion of Section 6.5 includes: 

• StdC prohibitions of commercial references, exclusive use of 
proprietary materials, or prescribing a proprietary agency for 
quality control or testing are met, and 

February 25,2011 

AL 

The revision to Section 6.7, 6.7.1 and 6.7.2 includes the addition of 
language regarding the criteria for project discontinuance. The previous 
sections 6.7 and 6.8 been renumbered due to this addition to 6.8 and 6.9 
respectively. 

February 25, 2011 

AM 

Section 6.10 was added, it includes the word writing to clearly specify the 
method in which interpretation requests are received and responded to. It 
also editorially corrects the spelling of the word revision. 

February 25,2011 

AN 

The revision of Section 8 deletes the entire paragraph and adds a blanket 
statement “ASHRAE agrees to comply with the Patent Policy as stated in 
the ANSI Essential Requirements.” 

February 25,2011 

AO 

The revision of Section 9 deletes the entire paragraph and adds a blanket 
statement “ASHRAE agrees to comply with the Commercial Terms and 
Conditions Policy as stated in ANSI Essential Requirements.” 

February 25, 2011 

AP 

This revision adds a Section 10 which includes information regarding 

PINS. It states” At the initiation of a project to develop or revise and 
ASHRAE American National Standard, ASHRAE shall use the ANSI 
Project Initiation Notification System (PINS) form. 

February 25,2011 
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AQ 

The revision to Annex A includes deletions of definitions. Deleted 
definitions include: 

• ASHRAE Alternate - a designated alternate to the ASHRAE 
Representative appointed by the Standards Committee of another 
organization and empowered to vote on behalf of ASHRAE on 
matters dealing with standards. (See ASHRAE Representative) 

• ASHRAE Representative - an official representative of ASHRAE 
appointed by the Standards Committee to a committee of another 
organization and empowered to vote on behalf of ASHRAE on 
matters dealing with standards. 

• Contact information - name, affiliation, mailing address, email 
address, daytime telephone numbers and facsimile numbers 

• Independent substantive change - a substantive change that is 
independent of any other substantive change and that does not 
significantly affect any other requirement in the standard. See 
substantive change. 

• Method of Test Standard - a standard setting forth the methods of 
measuring capacity or other characteristics of a specified material, 
component, or system, together with a specification of 
instrumentation, procedure, and calculations. 

• TC Technical Committee appointed by the TAC 

February 25,2011 

AR 

The revision to Section A2 includes the addition of the terms below: 

• BOD Board of Directors 

• PPIS Planning, Policy and Interpretations Subcommittee 

• SCD Standards Committee Document 

• SPLS Standards Project Liaison Subcommittee 

• SRS Standards Reaffirmation Subcommittee 

• TPS Title, Purpose and Scope 

The revision to Section A2 also includes a deletion of the terms below: 

• TC/TG/TRG a TC, TG or TRG 

• TC Technical Activities Committee 

February 25, 2011 

AS 

The revision to Section 6.2.1 removes one of the approving bodies, 
Technology Council. 

February 25, 2011 

AT 

The revision to Section 6.2.1.2 removes Technology Council and clarifies 
comment resolution attempts. It also notes that comments received that are 
not relevant to the proposed standards action under consideration shall be 
treated as a new proposal. 

February 25,2011 

AU 

The revision to Section 6.5 ensures that all procedures were followed and 
it provides the procedures for documenting consensus. 

February 25, 2011 

AV 

This revision adds a sentence to Section 10 which states: Comments will 
be addressed in accordance with clause 2.5 of the current version of the 
ANSI ER. 

February 25,2011 

AW 

The revision to Annex A adds definitions for informative language and 
notes. It also updates the current definitions; continuous maintenance, 
informative annex, normative annex, shall, should, standard, and 
unresolved public review commenter. 

February 25,2011 

AX 

The revision to Section B9.1 decreases the notice time to 30 days for 
appeal hearings if the appeal hearing is being held via conference call. 

February 25,2011 
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AY 

The revision to Section 4.2.1.1makes PASA consistent with ASHRAE’s 
Project Committee Manual of Procedures. 

May 15, 2012 

AZ 

The revision to Section 6.3.5 makes PASA consistent with ANSI ER. 

May 15, 2012 

BA 

The revision to Annex B would provide a larger pool of members to 
expeditiously hear appeals. 

May 15, 2012 

BC 

The revision to Section 6.2.1.2 brings PASA in line with StdC MOP and 
StdC Reference Manual. 

June 27,2012 

BD 

The revision to Section 6.3.6 is a direct result from the ExSC comments 
during the last public review of PASA. Procedures are included from when 
a Project Committee makes substantive changes to the draft after 
consideration of comments or when new information is received. 

June 27,2012 
(PASA Reaccredited 
10/12/12) 

BE 

Editorial change to Section 6.3.6.2, deleted last part of the sentence that 
states “in accordance with the continuous maintenance schedule.” 

September 27, 2013 

BF 

The revision to Section 4 adds additional information regarding Standards 
Subcommittees and its function as well as membership, most of this 
information was pulled from the PC MOP per ANSI’s request to 
streamline our documents. 

PASA Reaccredited 
October 22, 2014 

BG 

The revision to Section 7.2.1 adds information regarding the different types 
of Public Review and the publication approval level requirements. Section 
7.2.4 also clarifies the voting requirements for Standards Actions, 7.4.2 
and 7.4.3 clarifies lack of dominance and balance and interest categories. 
Section 7.6 clarifies criteria for approval. Section 7.8 allows SPLS and 

StdC to approve waivers for discontinuing a project. Section 7.11 supplies 
additional guidance for interpretation requests. PPIS can approve 
interpretations to the Standards Development Procedures. 

PASA Reaccredited 
October 22, 2014 

BH 

Annex A and A2 was revised to include additional definitions. Annex B 
was revised to clarify appealable matters, content of the appeal, filing fee, 
notification procedures, and conflict of interest. Annex C (Complaints of 
Actions or Inactions by the StdC, its Subcommittees or PC’s) and Annex 

E (Emergency Interim Standards Action) were added into PASA. 

PASA Reaccredited 
October 22, 2014 

Bl 

Annex A - editorial updates were made to the definitions: notes and code 
language document. “Notes” is now “Informative Notes”. 

November 10, 2014 

BJ 

Section 4.2.2.6 clarified SRS will comply with ANSI requirements of 
openness, balance and due process. Section 7.11 adds the Chair’s designee 
can also issue official interpretations of standards. Section 11 Antitrust 
Policy was added to PASA. Annex A, informative notes was clarified. 
Annex B removes the option for technical appeals. 

PASA Reaccredited 
April 29, 2015 

BK 

Section 7.4.4.1 was editorially corrected to mirror Annex B. (removes 
technical appeals) 

PASA - editorial 
September 3, 2015 



PASA - Procedures for ASHRAE Standards Actions 
ANSI Approved: April 29,2015 
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FORM FOR PROPOSALS FOR 2011 NATIONAL ELECTRICAL CODE® 


INSTRUCTIONS —PLEASE READ CAREFULLY 


FOR OFFICE USE ONLY 

Type or print legibly in black ink. Use a separate copy for each proposal. Limit 


Log- 

each proposal to a SINGLE section. All proposals must be received by NFPA bv 


5 p.m., EST, Friday, November 7, 2008. to be considered for five 201 i National 

Electrical Code. Proposals received after 5:00 p.m.. EST. Friday, November 7, 2008. 
will be returned to the submitter. If supplementary' material (photographs, diagrams, reports, 
etc.) is included, you may be required to submit sufficient copies for all members and 
alternates of the technical committee. 


Date Rec’d: 


Please indicate in which format you wish to receive your ROP/ROC C] electronic Q paper Q download 
(Note: If choosing the download option, you must view the ROP/ROC from our website; no copy will be sent to you.) 


Date Name Tel. No. 

Company 

Street Address City State Zip 

Please indicate organization represented (if any) 

1. Section/Paragraph 

2. Proposal Recommends (check one): new text 1 I revised text □ deleted text 

3. Proposal (include proposed new or revised wording, or identification of wording to be deleted): [Note: Proposed text should be in 
legislative format; i.e., use underscore to denote wording to be inserted (inserted wording ) and strike-through to denote wording to be deleted 
( d e leted wording ).] 


4. Statement of Problem and Substantiation for Proposal: (Note: State the problem that would be resolved by your recommendation; give the 
specific reason for your Proposal, including copies of tests, research papers, fire experience, etc. If more than 200 words, it may be abstracted for 
publication.) 


5. Copyright Assignment 

(a) d| I am the author of the text or other material (such as illustrations, graphs) proposed in this Proposal. 

(b) O Some or all of the text or other material proposed in this Proposal was not authored by me. Its source is as follows (please 
identify which material and provide complete information on its source): 


I agree that any material that I author, either individually or with others, in connection with work performed by an NFPA Technical Committee shall be considered to 
be works made for hire for the NFPA. To the extent that I retain any rights in copyright as to such material, or as to any other material authored by me that / submit for 
the use of an NFPA Technical Committee in the drafting of an NFPA code, standard, or other NFPA document. 1 hereby! grant and assign alt and full rights in 
copyright to the NFPA. I further agree and acknowledge that I acquire no rights in any publication of the NFPA and that copyright and all rights In materials 
produced by NFPA Technical Committees are owned by the NFPA and that the NFPA may register copyright in its own name. 


Signature (Required) 


PLEASE USE SEPARATE FORM FOR EACH PROPOSAL • NFPA Fax: (617) 770-3500 
Mail to: Secretary, Standards Council, National Fire Protection Association, 1 Batterymarch Park, Quincy. MA 02169-7471 


7/17/2007 
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FORM FOR PROPOSALS FOR 2008 NATIONAL ELECTRICAL CODE ® 


Mail to: Secretary, Standards Council 

National Fire Protection Association 
I Batterymarch Park, P.O. Box 9101 
Quincy, Massachusetts 02169-7471 
Fax to: (617) 770-3500 

Notes: 1. All proposals must be received by 5:00 p.m. EST on Friday, November 4, 2005. 

Proposals received after 5:00 p.m. EST, Friday, November 4, 2005, will be returned to the submitter. 

2. Type or print legibly in black ink. Limit each proposal to a SINGLE section, l.'se a separate copy for each proposal. 

3. If supplementary material ( photographs, diagrams, reports, etc.) is included, you may be required to submit 
sufficient copies for all members and alternates of the technical committee. 


FOR OFFICE USE ONLY 

Log#_ 

Date Rec’d_ 


Please indicate in which format you wish to receive your ROP/ROC: □ electronic □ paper □ download 


Date _ Name_Tel. No.:_ 

Company _ 

Street Address _ 

Organization Represented (if any)_ 

1. Section/Paragraph _ 

2. Proposal Recommends (check one) □ new text □ revised text □ deleted text 

3. Proposal (include proposed new or revised wording or identify wording to be deleted). Note: Proposed text 
should be in a legislative format: i.e., use underscore to denote wording to be inserted ( inserted wording ) and strike-through to denote 
wording to be deleted (de leted - word i ng) . 


4. Statement of Problem and Substantiation for Proposal. Note: State the problem that will be resolved by your recommendation; 
give the specific reason for your proposal and include copies of the tests, research papers, fire experience, etc. If more than 200 words, 
it may be abstracted for publication. 


5. LI This Proposal is original material. Note: Original material is considered to be the submitter’s own idea based on or as a 
result of his/her own experience, thought, or research and, to the best of his/her knowledge, is not copied from another source. 

□ This Proposal is not original material; its source (if known) is as follows: _ 


If you need further information on the standards-making process, please contact the 
Standards Administration Department at (617) 984-7249. 

For technical assistance, please call NFPA at (617) 770-3000. 


I hereby grant the NFPA all and full rights in copyright, in this proposal, and 1 understand- that l acquire no rights in any 
publication of NFPA in which this proposal in this or another similar or analogous form is used. 


Signature (required) 

PLEASE USE SEPARATE FORM FOR EACH PROPOSAL 
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Pace, John </0=ASTM/OU=FIRST ADMINISTRATIVE 
0m: GROUP/CN=RECIPIENTS/CN=JPACE> 

Sent: Tuesday, March 24, 2009 5:29 AM 

To: Hooper, Kathe <khooper@astm.org> 

Subject: Fw: Question related to copyright 


Fyi... I thought I had cc'd you! 


-Original Message- 

From: Pace, John 

To: FRVANBUREN@dow.com <FR VANBU REN@dow,com> 

Sent: Mon Mar 23 17:49:07 2009 
Subject: RE: Question related to copyright 

Dear Mr. Van Buren: 

I am responding on behalf of Kathe Hooper as she oversees all licensing and special permissions requests for ASTM International. 


First. Kathe has correctly stated ASTM Organizational policy: ASTM does not allow the posting of any of our copyrighted standards 
or other intellectual properties on the open Internet for possible free access or download. 


Second, the Disclaimer and Copyright notice of the European Patent Office does not provide sufficient protection nor use restraint if 
we allowed such such a request. There is no definition or limit as to who may access or download the copyrighted information from 
the EPO website, and there is no "click thru" license agreement addressing w ho would assume liability on further dow nstream use and 
control of the ASTM intellectual property. 


If DOW w ishes to assume responsibility to include lost revenues incurred by ASTM from such free posting, we can arrange with 
DOW and the EPO to have posted on this site a cover page of the standard with the abstract and metadata, and a link whereby any 
individual who needs a copy may obtain the pdf version via a click thru agreement, and the resulting pdf standard version dow nload 
will come directly from the ASTM server. For such downloads, ASTM would keep record and cltaige DOW on all copies 
downloaded on a monthly basis until the arrangement was officially terminated. 


If you w ish to pursue this arrangement, we will be more titan w illing to cooperate and w ork with you. 

Best Regards- 
John Pace 


John Pace 

Vice President. Publications and Marketing 

ASTM International 

610-832-9632 

jpace@astm.org 
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Dear Mrs. Hooper. 


Your refusal is regrettably difficult to accept for us. Therefore 1 will provide you with some additional explanation on the factual 
situation. 


We as an opponent in an European patent opposition are obliged to provide the documents mentioned in our notice of opposition as a 
hardcopy. Otherwise the opposition board of the European Patent Office will not consider the document. So this is one of the 
responsibilities of Dow in an European Patent Opposition. 


As mentioned before the documents are placed by the European Patent Office on a public website (European patent oppositions are 
essentially of public nature). How ever, the following is explicitly mentioned at this section of the EPO w ebsite (section in red by me): 


Disclaimer and copyright 


The Online File Inspection service gives users access to the infonnation contained in the European Patent Office (EPO) databases 
connected to the service. The EPO cannot assume liability for the correctness, completeness or quality' of the information thus 
accessed, nor can it guarantee that it is up to date. Documents viewed via this serv ice, particularly' non-patent literature items, may be 
subject to copyright. Before copying or using such documents in other electronic or printed publications, it is up to users of the Online 
Public File Inspection service to clieck w hether the permission of the author, publisher or other right holder is required. Where no 
third-party rights exist or are affected, the EPO gives permission for the infonnation retrieved to be reproduced together with an 
indication of the source, provided that live content is correctly reproduced. 


So the EPO has explicitly included this copyright notice. 


The step of submitting supporting infonnation by the opponent at the EPO is uncoupled from the responsibilities of the EPO and 
visitors of this section of the EPO website. The copyright aspects of downloading information from the website of the EPO have been 
addressed by the EPO. 


I hope this additional infonnation w ill allow' you to provide me with information how to obtain your permission for supplying the EPO 
with the nccessaiy copy of the ASTM standard. Please do not hesitate to contact me if you would like to have additional infonnation. 
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Frits van Buren 


Dr. F.R. van Buren 
Intellectual Capital Management 
PTC-1 / 439 building - office 103 
Dow Benelux B.V. 

P.O. Box 48 
4530 AA Temeuzen 
The Netherlands 

T + 31 115 672372 -F +31 115 673315 
fnanburcnWdovv.com 
Handelsregistemr. 24104547 


From: Hooper. Kathe | nia il t 9 ;khoQ p cr// . astnvQr g| 
Sent: Monday, March 16, 2009 8:14 PM 
To: Van Buren. Frederik (FR) 

Subject: RE: Question related to copyright 

Dear Mr. van Buren: 


This is in response to your email of 12 March (copy below). 


We are unable to grant permission as ASTM policy does not permit the posting of ASTM standards on public websites. 


Kindest regards, 

Kathe Hooper (Mrs.) 

ASTM International 

100 Barr Harbor Drive. PO Box C700 

West Conshohockea PA 19428-2959 

phone: 610-832-9634 

fax: 610-832-9635 
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email: khooper@astm.org 


From: Van Buren, Frederik (FR) 1 mailto :FRVANBUREN @dow.coml 
Sent: Thursday. March 12.2009 10:43 AM 
To: Custserv 

Subject: Question related to copyright 
Dear Mrs. or Mr.. 

March 03, 2009 a colleague of me (J. Kirsting - Dow Chemical - USA) ordered standard ASTM 1238 - 85 from IHS. 

On this standard it is mentioned: 

Copyright ASTM International 

Provided by IHS under license with ASTM 

No reproduction or networking permitted without license from IHS. 

and 

Sold to: Dow Chemical, 01742693 
Not for resale, 2009/3/3 20:57:9 GMT 

For an European patent opposition we need to file a hardcopy of this standard at the European Patent Office (EPO) in Munich in 
Germany. The documents filed at an opposition are placed on a public section of the website of the EPO as pdf files. There they can 
be red and downloaded. 

I would like to have your permission for filing a copy of this standard at the European Patent Office in Munich. 

I first submitted this question at IHS, but they referred to you for further information. 

Kind regards. 

Frits van Buren 
Dr. F.R. van Buren 

Terneuzen Intellectual Capital Management 

PTC-1 / 439 building - office 103 

Dow Benelux B.V 

P.O. Box 48 

4530 AA Terneuzen 

The Netherlands 

T + 31 115 672372 - F + 31 115 673315 
frvanburenWdow. com 
Handelsregistemr. 2410454 
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Hooper, Kathe </0=ASTM/OU=FIRST ADMINISTRATIVE 
from: GROUP/CN=RECIPIENTS/CN=KHOOPER> 

Sent: Thursday, July 9, 2009 3:33 PM 

To: 'Victor Palacios' <vic_3@hotmail.com> 

Subject: RE: Request (nao) 


Dear Mr. Palacios: 

Thank you for all your email and the information provided. 

After further review of your request, ASTM is unable to grant permission to reproduce ASTM standards B584 
and B208 in your thesis. You may reference the standards (by designation number and title) and refer readers 
to the ASTM website (www.astm.ora ) where they may purchase the standards. 

Thank you for your interest in ASTM standards. 

Kind regards, 


Kathe Hooper 

ASTM International 

100 Barr Harbor Drive, PO Box C700 

West Conshohocken, PA 19428-2959 

phone: 610-832-9634 

fax: 610-832-9635 

email: khooper@astm.org 


From: Victor Palacios [mailto:vic_3@hotmail.com] 

Sent: Wednesday, July 08, 2009 5:38 PM 
To: Hooper, Kathe 
Subject: RE: Request (nao) 

Dear Mrs. Kathe Hooper: 

My mailing address is: 

"Talleres Unidos Cevalios" 

Eloy Alfaro 1702 y Argentina 
Guayaquil, Ecuador 
Postal Code: EC090101 

Please be so kind to let me know the fees I need to cancel and all the information about the money transfer. 
Thank you very much for all your help. 

Kind regards, 

Victor Palacios 


De: Hooper, Kathe [mailto:khooper@astm.org] 

Enviado el: miercoles, 08 de julio de 2009 15:28 
Para: Victor Palacios 
Asunto: RE: Request (nao) 

Dear Victor, 

Please send your complete mailing address for the license agreement. 
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Also, please note that the license will give permission to make up to 5 copies (only) of the ASTM standards. 
No further reproduction of the ASTM standards (in full or in part) is permitted at University libraries or other 
places. 

Kind regards, 

Kathe Hooper 

ASTM International 

100 Barr Harbor Drive, PO Box C700 

West Conshohocken, PA 19428-2959 

phone: 610-832-9634 

fax: 610-832-9635 

email: khooper@astm.org 


From: Victor Palacios [mailto:vic_3@hotmail.com] 

Sent: Monday, July 06, 2009 4:54 PM 
To: Hooper, Kathe 
Subject: RE: Request (nao) 

Dear Mrs. Kathe Hooper: 

The digital copies will be delivered in CD-ROM. 

Thanks for your help. 

Victor Palacios 

De: Hooper, Kathe [mailto:khooper@astm.org] 

Enviado el: lunes, 06 de julio de 2009 14:52 
Para: Victor Palacios 
Asunto: RE: Request (nao) 

Dear Mr. Palacios, 

Thank you for your response. I have an additional question regarding the digital copies (PDF). How will you 
deliver the PDF files (i.e. CD-ROM, DVD?) 

Thank you. 

Kind regards, Kathe 


Kathe Hooper 

ASTM International 

100 Barr Harbor Drive, PO Box C700 

West Conshohocken, PA 19428-2959 

phone: 610-832-9634 

fax: 610-832-9635 

email: khooper@astm.org 


From: Victor Palacios [mailto:vic_3@hotmail.com] 
Sent: Monday, July 06, 2009 2:27 PM 
To: Hooper, Kathe 
Subject: RE: Request (nao) 
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Dear Mrs. Kathe Hooper: 

I need to make 1 original document (printed), 3 printed copies and two digital copies (in pdf format as part of the thesis so 
it can't be reproduced), that's 4 printed copies and two digital copies. These documents will be distributed as follows: 

I copy stays with the thesis director, 

1 copy and I original digital copy for the Mechanical Engineering library 
1 original, I copy and I original digital copy for the Central Campus Library 

I hope this information is the one you need, thanks in advance for all your help, 

Victor Palacios 

De: Hooper, Kathe [mailto:khooper@astm.org] 

Enviado el: lunes, 06 de julio de 2009 10:24 
Para: vic_3@hotmail.com 
Asunto: RE: Request (nao) 

Dear Mr. Palacios: 

This is in regard to your email of 1 July (copy below). 

Before we can proceed with your request to include ASTM standards B584 and B208 in your thesis, we will 
need to know how many printed copies of your thesis will be made and distributed. Once we receive this 
information, we will be happy to send a license agreement outlining the fees and conditions involved. 

Please note that ASTM policy requires a fee for the rights to reproduce and distribute printed copies of ASTM 
standards. Also, ASTM does not permit the posting of ASTM standards on public websites or the distribution 
of the PDF files. 

Thank you for your interest in ASTM standards. 

Kind regards, 


Kathe Hooper (Mrs.) 

ASTM International 

100 Barr Harbor Drive, PO Box C700 

West Conshohocken, PA 19428-2959 

phone: 610-832-9634 

fax: 610-832-9635 

email: khooper@astm.org 


From: Naouri, Sarah 
Sent: Thursday, July 02, 2009 9:21 AM 
To: Hooper, Kathe 
Cc: srvcout 

Subject: FW: Request (nao) 

Hi Kathe, 

Sorry for all the emails today! Would the below permission request be something you handle? Please advise. Thank you. 

Best Regards, 

Sarah Naouri 

ASTM International 

Customer Relations Representative 
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From: Custserv 

Sent: Thursday, July 02, 2009 8:45 AM 
To: Naouri, Sarah 
Subject: FW: Request 


From: Victor Palacios [mailto:vic_3@hotmail.com] 

Sent: Wednesday, July 01,2009 3:46 PM 
To: Custserv 
Subject: Request 

Greetings, 

My name is Victor Palacios, I'm from Ecuador and I bought two standards: B584 and B208 through a friend's credit card 
(Jose Eduardo Rossel) two years ago (approximately). I'm making a thesis for my degree in Mechanical Engineering. The 
thesis is about the fabrication of copper alloy casting C86500 according to the AS7M Standard B584 for marine 
applications. The reason I write this email is because I would like your authorization to use these standards as annex 
documents in the thesis. Obviously I can't publish them without your authorization. 

As I said, my name is Victor Palacios Cevallos, the university I studied is Escuela Superior Politecnica del Litoral (ESPOL), my 
thesis director is Ignacio Wiesner Falconi (email: iwiesner@espol.edu.ec), Mech. Eng. The university's web page is 
www.espol.edu.ee. the faculty's web page is www.fimcp.espol.edu.ee . 

If there is a formal procedure of doing this, please let me know. 

Thanks in advance, 

Victor Palacios 
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REGISTER MY ACCOUNT 


Password Note: Please use only letters A(a)-Z(z) and numbers 0-9. Do not use any punctuation 
marks, formatting, spacing, or non-letters (periods, commas, etc.). Your password must be at least 6 
characters long. 


All Search topic, title ,... 



E-Mail address: 
Select Password: 
Confirm Password: 


Home i About ASTM i Site Map I Support I Contact I Policies l Copyright/Permissions 

Copyright © 1996 - 2015 ASTM. All Rights Reserved. ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA, 19428-2959 USA 
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Your Address Recommended 


* First Name 
Middle Name 


* Last Name 


Please enter your customer contact information in the fields below. 


ASTM Training: 
Apply standards 
more effectively 


* Organization / Company 


* Street Address 


Check here if you do not have a company affiliation 



Train at our location 
or yours, and get 
instruction on the 
most important 
standards you use 


City 
* Country 
State 
* Postal Code 

* Phone 

* Email 

□ 


Pleas.. 

Selec.. 



mstoltz777@yahoo.com 



CONTINUE 


All Search topic, title .... 


Home l About ASTM I Site Map I Support I Contact I Policies I Copyright/Permissions 

Copyright 01996 - 2015 ASTM. All Rights Reserved. ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA, 19428-2959 USA 
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Recommended 


This is a service where you can view and read ASTM safety standards incorporated in United States 
regulations. 

The standards are presented for online reading. There are no print or download options. To acquire 
the current version of the standards with print and download options, follow the hyperlink to the 
current version. 


CAUTION: The ASTM standards available on this site are the versions and year-dates actually 
referenced in the respective federal regulation or law. The ASTM standard referenced MAY NOT BE 
THE MOST RECENT OR UP-TO-DATE version available. It is possible that the standard and/or 
technology at issue has changed or been updated during the period of time since the regulation/law 
was enacted. As a non-governmental organization, ASTM does not control which ASTM standards 
(and versions thereto) are referenced in federal regulations or laws. 


Send comments or questions to servicetaastm.org 
OPEN READING ROOM 


& 

ASTM Proficiency 
Testing: improve 
your lab's 
performance 

Meet accreditation 
requirements, 
compare your 
performance with 
other labs, 
document your 
expertise. 


All Search topic, title ,. 


Home i About ASTM i Site Map I Support I Contact I Policies I Copyright/Permissions 
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The purpose of this site is to provide the public with access to ASTM International standards 
(“ASTM Documents”) which have been referenced or incorporated into federal regulation or 
laws. Please use this site to review these standards. The ASTM Documents are provided as a 
public service, and you represent that you will not make any commercial use of the ASTM 
Documents available here. These ASTM Documents are available for review only, and 
hardcopies and printable versions will continue to be available for purchase. By clicking on any 
ASTM Document, you agree to be bound by the terms of this agreement both as to this and each 
subsequent use you make of the ASTM Document, and you are responsible for ensuring that the 
terms of this agreement are met. 

IMPORTANT- READ THESE TERMS CAREFULLY BEFORE ACCESSING ANY ASTM 
DOCUMENT. By accessing any ASTM Document you are entering into a contract, and 
acknowledge that you have read this License Agreement, that you understand it and agree to be 
bound by its terms. If you do not agree to the terms of this License Agreement, promptly exit this 
site. 

License: ASTM grants you, the ASTM visitor, a nonexclusive and nontransferable license to 
view online the content of the ASTM Document(s). The ASTM Document is designed to be 
viewed online only - there are no “print,” “save,” or “cut and paste” options - and the license 
granted to you by this agreement does not include the right to download, reproduce, store in a 
retrieval system, modify, make available on a network, use to create derivative works, or 
transmit the content of the ASTM Document in any form or by any means, electronic, 
mechanical, photocopying, recording, scanning, or otherwise. 

This license is specifically granted conditioned on your completion of the on-line 
registration form and you represent that the information you provided is truthful and 
accurate. 

Copyright: This site and all of its content are protected by copyright pursuant to U.S. and 
international copyright laws. You may not copy or download any of the material contained on 
this site in whole or in part without the express authorization of ASTM. You may not publish, 
modify, transmit, reproduce, create new works from, distribute, sell, loan, nor in anyway exploit 
any of the material contained on this site in whole or in part, without the express authorization of 
ASTM. 

Trademark: Except as indicated, ASTM owns all trademarks, service marks, certification 
marks, and logos featured on this site, including the terms "ASTM," ASTM International” and 
the "American Society for Testing and Materials." Use of these marks without the express 
written permission of ASTM is expressly prohibited. 

Indemnification: You agree to indemnify and hold ASTM, its directors, officers, members, 
and employees harmless from any claims, demands, or damages, including attorney fees, 
asserted by any third party due to or arising out of your use of or conduct on the site or of any 
ASTM Document. 

Disclaimer of Warranty and Liability: ASTM MAKES NO REPRESENTATION THAT 
THE DOCUMENTS ON THIS SITE ARE THE MOST RECENT OR UP-TO-DATE VERSION 
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OF THE ASTM STANDARDS CURRENTLY AVAILABLE. IT IS THE VISITOR’S 
RESPONSIBILITY TO DETERMINE IF THE DOCUMENT MEETS THEIR 
REQUIREMENTS OR PURPOSES. 

ASTM SHALL NOT BE LIABLE FOR ANY DIRECT, SPECIAL, INDIRECT, INCIDENTAL, 
CONSEQUENTIAL, OR PUNITIVE DAMAGES, INCLUDING WITHOUT LIMITATION, 
LOST REVENUES OR LOST PROFITS, WHICH MAY RESULT FROM THE USE OF, 
ACCESS TO, OR INABILITY TO USE THESE MATERIALS. UNDER NO 
CIRCUMSTANCES WILL THE TOTAL LIABILITY OF ASTM TO YOU BASED ON ANY 
CAUSE OF ACTION EXCEED $100. 

Miscellaneous: As a condition of your use of this site, you agree not to use the site for any 
purpose that is unlawful or prohibited by this agreement. 

Use of the site by you is unauthorized in any jurisdiction that does not give effect to all 
provisions contained in this agreement. 

If any part of these terms and conditions is held to be invalid or unenforceable for any reason 
including, but not limited to, the warranty disclaimers and liability limitations specified above, 
then the invalid or unenforceable provision will be deemed superseded by a valid enforceable 
provision that most closely matches the intent of the original provision and the remainder of the 
agreement will remain in full force and effect. 

A printed version of this agreement shall be admissible injudicial or administrative proceedings 
based upon or relating to this agreement to the same extent and subject to the same conditions as 
other business documents and records originally generated and maintained in printed form. 

These terms and conditions constitute the entire agreement between you and ASTM with respect 
to your use of the site. You acknowledge that, in providing you access to and use of the site, 
ASTM has relied on your agreement to be legally bound by these terms and conditions. 

This agreement shall be construed and interpreted pursuant to the laws of the Commonwealth of 
Pennsylvania applicable to agreements wholly entered into and performed in Pennsylvania, 
excluding that body of law dealing with conflict of laws. Any legal action, suit, or proceeding 
arising out of or relating to this agreement or the breach thereof shall be instituted in a court of 
competent jurisdiction in Pennsylvania, and each party hereby consents and submits to the 
personal jurisdiction of such court, waives any objection to venue in such court and consents to 
the service of process by registered or certified mail, return receipt requested, at the last known 
address of such party. 

You may not assign or transfer your rights or obligations under this agreement. 
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The purpose of this site is to provide the public with access to ASTM International standards (“ASTM 
Documents”) which have been referenced or incorporated into federal regulation or laws. Please use 
this site to review these standards. The ASTM Documents are provided as a public service, and you 
represent that you will not make any commercial use of the ASTM Documents available here. These 
ASTM Documents are available for review only, and hardcopies and printable versions will continue 
to be available for purchase. By clicking on any ASTM Document, you agree to be bound by the 
terms of this agreement both as to this and each subsequent use you make of the ASTM Document, 
and you are responsible for ensuring that the terms of this agreement are met. 

IMPORTANT- READ THESE TERMS CAREFULLY BEFORE ACCESSING ANY ASTM DOCUMENT. 

By accessing any ASTM Document you are entering into a contract, and acknowledge that you have 
read this License Agreement, that you understand it and agree to be bound by its terms. If you do not 
agree to the terms of this License Agreement, promptly exit this site. 


License: 

ASTM grants you, the ASTM visitor, a nonexclusive and nontransferable license to view online the 
content of the ASTM Document(s). The ASTM Document is designed to be viewed online only - there 
are no “print,” “save,” or “cut and paste" options - and the license granted to you by this agreement 
does not include the right to download, reproduce, store in a retrieval system, modify, make available 
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Please indicate your acceptance of the following terms for accessing the NFPA online 
document you have selected (“the Online Document”) by scrolling down the page and 
clicking "I AGREE" to connect. By clicking on "I AGREE," you accept the terms of this 
agreement 

This is a legal agreement between you (the NFPA visitor) and the NFPA for access to 
and use of the Online Document. By clicking on “I AGREE” below and by using the 
Online Document, you agree to be bound by the terms of this agreement both as to this 
and each subsequent use you make of the Online Document, and you are responsible 
for ensuring that the terms of this agreement are met. If you do not agree to the terms of 
this agreement, click on the “Return to the Home Page” button below. 

GRANT OF LICENSE. NFPA grants you, the NFPA visitor, a nonexclusive and 
nontransferable license to view online the content of the Online Document. The Online 
Document is designed to be viewed online only - there are no “print,” “save,” or “cut and 
paste” options - and the license granted to you by this agreement does not include the 
right to download, reproduce, store in a retrieval system, modify, make available on a 
network, use to create derivative works, or transmit the content of the Online Document 
in any form or by any means, electronic, mechanical, photocopying, recording, 
scanning, or otherwise. 

COPYRIGHT. You acknowledge that the content of the Online Document is 
copyrighted and owned by NFPA and is protected by U.S. copyright law and 
international treaty provisions. You acquire no proprietary interest in the Online 
Document or any of the information displayed therein. Nothing herein is intended to 
prohibit you from making limited, non-commercial use of the content of any NFPA 
codes, standards, guides, and recommended practices to the extent that such use is a 
“fair use” under the copyright laws of the United States. However, such fair use does not 
include the disabling, circumventing, or otherwise evading the read-only or other 
technological measures that limit copying of the content of the Online Document. No 
copying beyond that permitted by “fair use” shall be permitted without the express 
written permission of the NFPA. Permission will be considered based on a written 
request to the Associate General Counsel, NFPA, 1 Batterymarch Park, P.O. Box 9101, 
Quincy, MA 02269-9101. Print and full-featured electronic versions of NFPA codes, 
standards, guides, and recommended practices, including the Online Document, are 
available for purchase through this web site or by contacting NFPA at 1-800-344-3555. 

WARRANTY LIMITATION AND DISCLAIMER OF LIABILITY. THE ONLINE 
DOCUMENT IS PROVIDED TO YOU “AS” IS” AND WITHOUT WARRANTIES OF ANY 
KIND, EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATIONS THE IMPLIED 
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 
PURPOSE. NFPA SHALL NOT BE LIABLE FOR ANY DAMAGE OR LOSS OF ANY 
KIND ARISING OUT OF, RESULTING FROM, OR IN ANY WAY RELATED TO THE 
ONLINE DOCUMENT, INCLUDING WITHOUT LIMITATION, (A) ANY ERRORS IN OR 
OMISSIONS IN THE CONTENT OF THE ONLINE DOCUMENT; (B) THE 
UNAVAILABILITY OR INTERRUPTION OF ACCESS TO THE ONLINE DOCUMENT; 
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(C) YOUR USE OF ANY EQUIPMENT OR SOFTWARE IN CONNECTION WITH 
USING THE ONLINE DOCUMENT; AND (D) YOUR USE OF THE CONTENT AND 
INFORMATION OR OPINIONS CONTAINED IN THE ONLINE DOCUMENT. OTHER 
IMPORTANT NOTICES AND DISCLAIMERS OF LIABILITY ARE CONTAINED IN THE 
INTRODUCTORY MATERIALS INCLUDED AT THE BEGINNING OF ALL PRINT AND 
ELECTRONIC VERSIONS OF NFPA CODES, STANDARDS, GUIDES, AND 
RECOMMENDED PRACTICES, INCLUDING THE ONLINE DOCUMENT, AND ARE 
INCORPORATED HEREIN BY REFERENCE. SOME STATES RESTRICT 
WARRANTY AND REMEDY EXCLUSIONS AND LIMITATIONS, AND, TO THE 
EXTENT OF SUCH RESTRICTIONS, THE FOREGOING LIMITATIONS MAY NOT 
APPLY TO YOU. IN SUCH STATES, NFPA’S LIABILITY SHALL BE LIMITED TO THE 
GREATEST EXTENT PERMITTED BY LAW. 

MISCELLANEOUS. The terms of this agreement may be changed from time to 
time. NFPA may suspend or discontinue providing the Online Document to you 
with or without cause and without notice. NFPA may pursue any remedy legally 
available to it if you fail to comply with any of your obligations hereunder. The failure of 
NFPA to enforce any provision hereof shall not constitute or be construed as a waiver of 
such provision or of the right to enforce it at a later time. 

You may not assign or transfer your rights or obligations under this agreement. 

This agreement shall be construed and interpreted pursuant to the laws of the State of 
Massachusetts applicable to agreements wholly entered into and performed in the State 
of Massachusetts, excluding that body of law dealing with conflict of laws. Any legal 
action, suit, or proceeding arising out of or relating to this agreement or the breach 
thereof shall be instituted in a court of competent jurisdiction in the State of 
Massachusetts, Norfolk County, and each party hereby consents and submits to the 
personal jurisdiction of such court, waives any objection to venue in such court and 
consents to the service of process by registered or certified mail, return receipt 
requested, at the last known address of such party. 

The terms of this agreement constitute the entire agreement between the parties with 
respect to the subject matter hereof. If any provision hereof is adjudged to be invalid, 
void, or unenforceable, the parties agree that the remaining provisions hereof will not be 
affected thereby, that the provision in question may be replaced by the lawful provision 
that most nearly embodies the original intention of the parties, and that the terms of this 
agreement will in any event remain valid and enforceable. 
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series with the fluid flow, such as expansion tanks, fill lines, 
chemical feeders, and drains. 

plenum: a compartment or chamber to which one or more 
ducts are connected, that forms a part of the air distribution 
system, and that is not used for occupancy or storage. A ple¬ 
num often is formed in part or in total by portions of the 
building. 

pool: cny structure, basin, or tank containing an artificial 
body of water for swimming, diving, or recreational bathing. 
The term includes, but is not limited to, swimming pool, 
whirlpool, spa, and hot tub. 

power roof/wall ventilators (PRY): a fan consisting of a cen¬ 
trifugal or axial impeller with an integral driver in a weather- 
resistant housing and with a base designed to fit, usually by 
means of a curb, over a wall or roof opening. 

power usage effectiveness (PUE): computer room energy 
divided by IT equipment energy calculated in accordance with 
industry-accepted standards (see Informative .Appendix E). 
pover usage effectiveness—category 0 (PUE 0 JL: peak 
electric demand (kW) for the entire computer room, 
including IT equipment and supporting infrastructure, 
divided by peak electric demand (kW) of the IT equip¬ 
ment. 

pover usage effectiveness—category I (PUE/): annual 
energy consumption (kWh) for the entire computer room, 
including IT equipment and supporting infrastructure, 
divided by annual energy consumption (kWh) of the IT 
equipment. 


purchased energy rates: costs for units of energy or power 
purchased at the building site. These costs may include 
energy costs as well as costs for power demand as determined 
by the adopting authority. 

R-value: see thermal resistance. 

radiant heating system: a heating system that transfers heat 
to objects and surfaces within the heated space primarily 
(greater than 50%) by infrared radiation. 

rated motor power: see motor power, rated. 

rated R-value of insulation: the thermal resistance of the 
insulation alone as specified by the manufacturer in units of 
h-ft^’F/Btu at a mean temperature of 75°F- Rated R-value 
refers to the thermal resistance of the added insulation in 
framing cavities or insulated sheathing only and does not 
include the thermal resistance of other building materials or 
air films. (See thermal resistance.) 

rating authority: the organization or agency that adopts or 
sanctions use of this rating methodology. 

readily accessible: capable of being reached quickly for oper¬ 
ation, renewal, or inspection without requiring those to whom 
ready access is requisite to climb over or remove obstacles or 
to resort to portable ladders, chairs, etc. In public facilities, 
accessibility may te limited to certified personnel through 
locking cov ers or by placing equipment in locked rooms. 

recirculating system: a domestic or service hot-water distri¬ 
bution system tha: includes a closed circulation circuit 
designed to maintain usage temperatures in hot-water pipes 
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The purpose of this site is to provide the public with access to ASTM International standards (“ASTM Documents”) 
which have been referenced or incorporated into federal regulation or laws. Please use this site to review these 
standards. The ASTM Documents are provided as a public service, and you represent that you will not make any 
commercial use of the ASTM Documents available here. These ASTM Documents are available for review only, and 
hardcopies and printable versions will continue to be available for purchase. By clicking on any ASTM Document, 
you agree to be bound by the terms of this agreement both as to this and each subsequent use you make of the ASTM 
Document, and you are responsible for ensuring that the terms of this agreement are met. 

IMPORTANT- READ THESE TERMS CAREFULLY BEFORE ACCESSING ANY ASTM DOCUMENT. 

By accessing any ASTM Document you are entering into a contract, and acknowledge that you have read this License 
Agreement, that you understand it and agree to be bound by its terms. If you do not agree to the terms of this License 
Agreement, promptly exit this site. 

License: 

ASTM grants you, the ASTM visitor, a nonexclusive and nontransferable license to view online the content of the 
ASTM Document(s). The ASTM Document is designed to be viewed online only - there are no “print,” “save,” or 
“cut and paste” options - and the license granted to you by this agreement does not include the right to download, 
reproduce, store in a retrieval system, modify, make available on a network, use to create derivative works, or 
transmit the content of the ASTM Document in any form or by any means, electronic, mechanical, photocopying, 
recording, scanning, or otherwise. 

This license is specifically granted conditioned on your completion of the on-line registration form and you represent 
that the information you provided is truthful and accurate. 

Copyright: 

This site and all of its content are protected by copyright pursuant to U.S. and international copyright laws. You may 
not copy or download any of the material contained on this site in whole or in part without the express authorization 
of ASTM. You may not publish, modify, transmit, reproduce, create new works from, distribute, sell, loan, nor in 
anyway exploit any of the material contained on this site in whole or in part, without the express authorization of 
ASTM. 

Trademark: 

Except as indicated, ASTM owns all trademarks, service marks, certification marks, and logos featured on this site, 
including the terms "ASTM," ASTM International” and the "American Society for Testing and Materials." Use of 
these marks without tire express written permission of ASTM is expressly prohibited. 

Indemnification: 

You agree to indemnify and hold ASTM, its directors, officers, members, and employees harmless from any claims, 
demands, or damages, including attorney fees, asserted by any third party due to or arising out of your use of or 
conduct on the site or of any ASTM Document. 

Disclaimer of Warranty and Liability: 

ASTM MAKES NO REPRESENTATION THAT THE DOCUMENTS ON THIS SITE ARE THE MOST RECENT 
OR UP-TO-DATE VERSION OF THE ASTM STANDARDS CURRENTLY AVAILABLE. IT IS THE VISITOR’S 
RESPONSIBILITY TO DETERMINE IF THE DOCUMENT MEETS THEIR REQUIREMENTS OR PURPOSES. 

ASTM SHALL NOT BE LIABLE FOR ANY DIRECT, SPECIAL, INDIRECT, INCIDENTAL, 
CONSEQUENTIAL, OR PUNITIVE DAMAGES, INCLUDING WITHOUT LIMITATION. LOST REVENUES 
OR LOST PROFITS, WHICH MAY RESULT FROM THE USE OF, ACCESS TO, OR INABILITY TO USE 
THESE MATERIALS. UNDER NO CIRCUMSTANCES WILL THE TOTAL LIABILITY OF ASTM TO YOU 
BASED ON ANY CAUSE OF ACTION EXCEED $100. 

Miscellaneous: 

As a condition of your use of this site, you agree not to use the site for any purpose that is unlawful or prohibited by 
this agreement. 

Use of the site by you is unauthorized in any jurisdiction that does not give effect to all provisions contained ill this 
agreement. 
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If any pari of these terms and conditions is held to be invalid or unenforceable for any reason including, but not 
limited to, the warranty disclaimers and liability limitations specified above, then the invalid or unenforceable 
provision will be deemed superseded by a valid enforceable provision that most closely matches the intent of the 
original provision and the remainder of the agreement wall remain in full force and effect. 

A printed version of this agreement shall be admissible injudicial or administrative proceedings based upon or 
relating to this agreement to the same extent and subject to the same conditions as other business documents and 
records originally generated and maintained in printed form. 

These terms and conditions constitute the entire agreement between you and ASTM with respect to your use of the 
site. You acknowledge that, in providing you access to and use of the site, ASTM has relied on your agreement to be 
legally bound by these terms and conditions. 

This agreement shall be construed and interpreted pursuant to the laws of the Commonwealth of Pennsylvania 
applicable to agreements wholly entered into and performed in Pennsylvania, excluding that body of law dealing with 
conflict of laws. Any legal action, suit, or proceeding arising out of or relating to this agreement or the breach thereof 
shall be instituted in a court of competent jurisdiction in Pennsylvania, and each party hereby consents and submits to 
the personal jurisdiction of such court, waives any objection to v enue in such court and consents to the service of 
process by registered or certified mail, return receipt requested, at the last known address of such party. 

You may not assign or transfer your rights or obligations under this agreement. 
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1 

it. 

1 

beyond the document production to verify that 

2 

Objection to form. You're 

2 

information. 

3 

asking him to recall, without having 

3 

Q. But you don't recall seeing any 

4 

all the materials in front of him? 

4 

defective materials yourself, correct? 

5 

MR. BRIDGES: Yeah. 

5 

A. That's correct. I do not. 

6 

MR. FEE: Okay. 

6 

Q. You just relied upon the word 

7 

THE WITNESS: It's all laid out 

7 

of others, correct? 

8 

in my report, and the sources are 

8 

MR. FEE: Objection. Vague. 

9 

provided in my report. I've not 

9 

Mischaracterizes his testimony. 

10 

memorized all those. 

10 

THE WITNESS: I relied upon 

11 

BY MR. BRIDGES: 

11 

written documents I saw and 

12 

Q. But I don't think your report 

12 

conversations that I had. 

13 

refers to upside-down materials, does it? 

13 

BY MR. BRIDGES: 

14 

A. I don't recall for sure, but I 

14 

Q. What written documents did you 

15 

thought some of the documents that I cited 

15 

see that discussed these issues? 

16 

make reference to those materials. I'm not 

16 

MR. FEE: Objection. Asked and 

17 

sure that I cited the, for instance, 

17 

answered. 

18 

upside-down materials, but I think I have 

18 

THE WITNESS: And I'm sorry. I 

19 

discussions about that phenomenon. 

19 

can't point you to the particular 

20 

Q. With whom? 

20 

ones. Perhaps, through the course of 

21 

A. In written materials that I've 

21 

the day, my memory will be refreshed 

22 

cited. 

22 

on that. 

23 

Q. Have you had oral discussions 

23 

BY MR. BRIDGES: 

24 

about what you have referred to as that 

24 

Q. If you relied upon those 

25 

phenomenon? 

25 

written documents, would you have cited to 
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Page 24 

1 

A. Yes. 

1 

those written documents in your report? 

2 

Q. With whom? 

2 

A. Perhaps. 

3 

A. Counsel here. 

3 

Q. Why do you say "perhaps"? 

4 

Q. With anybody else? 

4 

A. I can't say with absolute 

5 

A. I don't think so. It's 

5 

certainty what I do. But often, if something 

6 

possible, but I'm not recalling anything 

6 

is a direct support for a factual 

7 

else. 

7 

observation, I will often cite that source, 

8 

Q. And when you say discussions 

8 

but not always. 

9 

with "counsel here," you're referring to the 

9 

Q. What previous - strike that. 

10 

counsel at the table here today at the 

10 

What training or education have 

11 

deposition? 

11 

you ever received with respect to standards 

12 

A. Correct. 

12 

development organizations? 

13 

And we should add to that 

13 

MR. FEE: Objection to form. 

14 

Jordana Rubel, who's been a person that I've 

14 

THE WITNESS: I don't recall if 

15 

had conversations with over the last several 

15 

I've had a course in standard 

16 

months. 

16 

development. Probably it has been 

17 

Q. What did you do to verify any 

17 

part of some of the economics courses 

18 

of the statements to you from counsel about 

18 

that I've taken over the years. 

19 

these facts you've referred to about the 

19 

In my profession and the work 

20 

materials that the defendant has 

20 

that I've done in the last 30 years, 

21 

disseminated? 

21 

I've had occasion to look at and 

22 

A. I don't think I did separate 

22 

evaluate standards organizations and 

23 

verification. I may have seen some documents 

23 

the output from those organizations. 

24 

that provide or provided confirmation of that 

24 

So it is among the topics that 

25 

fact, but I don't recall separately going out 

25 

I've investigated in the course of my 
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1 

consulting career. 

1 

standards development organization that 

2 

BY MR. BRIDGES: 

2 

you've worked on? 

3 

Q. In what context? 

3 

A. Again, I'd have to go back and 

4 

A. There have been several matters 

4 

look at my records. I can't right now recite 

5 

I've had, litigations, that have involved 

5 

any, but there very well could be one or 

6 

standard setting organizations and the 

6 

more. 

7 

outputs from those organizations. 

7 

Q. Did you review any of your work 

8 

Q. What organizations? 

8 

in — from earlier copyright cases involving 

9 

A. Well, some that come to mind 

9 

standards development organizations in 

10 

are ETSI, IEEE, the Blu-ray Association, 

10 

connection with your work in this case? 

11 

MPEG, MPEG L.A., the Philips 6C and Philips 

11 

A. Not to the best of my memory, 

12 

3C organizations. Those are among the ones 

12 

no. 

13 

that come to mind. 

13 

Q. What background do you have in 

14 

Q. And what types of litigation 

14 

the creation of standards by standard 

15 

did your work relating to those standard 

15 

development organizations? 

16 

setting organizations involve? 

16 

MR. FEE: Objection to form. 

17 

MR. FEE: Objection to form. 

17 

THE WITNESS: In the context of 

18 

THE WITNESS: It was almost all 

18 

some of my consulting assignments, I 

19 

intellectual property litigation, with 

19 

have examined processes undertaken by 

20 

probably the bulk of the analyses 

20 

SDOs. 

21 

undertaken with regard to patent 

21 

BY MR. BRIDGES: 

22 

rights. 

22 

Q. Anything else? 

23 

BY MR. BRIDGES: 

23 

A. Nothing else comes to mind. 

24 

Q. Do you recall — 

24 

I've certainly looked at the output 

25 

A. I guess I should — there were 

25 

associated with those processes, but there's 
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1 

probably some breach of contract matters as 

1 

nothing else that comes to mind. 

2 

well. 

2 

Q. What processes undertaken by 

3 

Q. Did you work on any matters 

3 

standards development organizations did you 

4 

involving copyright law where you became 

4 

examine? 

5 

familiar with the work and outputs of 

5 

MR. FEE: Objection. Are you 

6 

standards setting organizations before this 

6 

asking prior to the report still? 

7 

case? 

7 

MR. BRIDGES: Yes. 

8 

A. Probably, but I cannot say that 

8 

MR. FEE: Okay. 

9 

with absolute certainty. I've been involved 

9 

THE WITNESS: I'm not quite - 

10 

in several matters over a course of many 

10 

MR. BRIDGES: Or other than in 

11 

years. 

11 

this case. 

12 

Q. Can you name any copyright 

12 

MR. FEE: Okay. 

13 

matter involving a standards development 

13 

THE WITNESS: I'm not quite 

14 

organization that you recall? 

14 

sure what you're asking. I've seen 

15 

A. Not now, without going back and 

15 

discussion of the some of the 

16 

looking at my records. 

16 

processes of various organizations. 

17 

Q. Would they be listed in the 

17 

I'm not - I'm not quite sure what 

18 

cases attached to Exhibit 1 ? 

18 

you're asking. Perhaps you could ask 

19 

A. That would summarize some of my 

19 

it somewhat differently. 

20 

records. The cases that are embodied in my 

20 

BY MR. BRIDGES: 

21 

tab 1 are those that led to deposition or 

21 

Q. Well, no. You said, quote, "I 

22 

trial testimony. I've been involved in many 

22 

have examined processes undertaken by SDOs.' 

23 

matters beyond those. 

23 

So my question is, what 

24 

Q. But sitting here, you cannot 

24 

processes undertaken by standards development 

25 

recall any copyright case involving a 

25 

organizations did you examine? 
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1 A. It sounds like the same 

2 question to me. 

3 Q. Specifically, what processes 

4 did you examine? 

5 A. That still sounds like the same 

6 question, but let me try to answer it by 

7 saying I've looked, for instance, at the 

8 mechanisms that ETSI undertook in developing 

9 standards. So I am familiar generally with 

10 the processes that it follows. Similarly 

11 with regard to other standard setting 

12 organizations. 

13 Q. What other standard setting 

14 organizations? 

15 A. Well, I think I identified 

16 those a few moments ago. Do you want me to 

17 repeat those? 

18 Q. Well, if — are you saying 

19 that, for all of those organizations, you 

20 examined their processes? 

21 A. In some dimension, probably for 

22 most of the organizations, I had at least 

23 some knowledge of the process. I can't say 

24 that I investigated in depth all of the 

25 processes for all of the organizations that 

Page 30 

1 manufacturers only. Others include a wider 

2 array of companies. 

3 In all instances, though, the 

4 companies are trying to — the standards 

5 setting organizations are trying to develop 

6 at least some form of consensus - sometimes 

7 it's very broad consensus; sometimes it's 

8 more narrow consensus - about what would be 

9 good for that standards setting organization. 

10 Sometimes the SSOs are 

11 interested in what's best for the 

12 manufacturers and the ability for them to 

13 supply in an interoperable environment. In 

14 some cases, the SSOs are very alert to the 

15 needs of consumers and users of products and 

16 services that comply with standards. 

17 Q. You've distinguished between 

18 standards setting organizations and standard 

19 development organizations. What is the 

20 distinction that you — that you identify 

21 between the two? 

22 A. I think I said I didn't know if 

23 there is for sure a distinction, but I think 

24 an SSO is perhaps a broader concept than an 

25 SDO, but I might be wrong on that. 

Page 32 

1 have been involved in my consulting 

2 assignments that are standards oriented. 

3 Q. What do you recall about your 

4 investigation of the processes by which 

5 standards development organizations create 

6 their standards? 

7 A. I should say I - SDO is 

8 probably not the right term to use. I should 

9 probably say standards setting organizations. 

10 There may be a distinction between an SSO and 

11 an SDO. 

12 But, generally, each SSO has a 

13 process that's unique to its organization. 

14 Some solicit input from a wide range of 

15 constituents; some from a more narrow range. 

16 The ones that I have examined 

17 have all been fairly careful in the work that 

18 they've done, seeking input at many steps 

19 along the way. 

20 Some organizations, like SDOs 

21 at issue here, seek a broader array of inputs 

22 than do others. 

23 Some organizations, standards 

24 setting organizations, include primarily or 

25 only manufacturers and sometimes large 

Page 31 

1 I know the companies — I - 

2 the plaintiffs here are SDOs. The 

3 associations are, among other things, in the 

4 business of creating and developing 

5 standards. 

6 There could be other SSOs that 

7 have different constituents that are of 

8 interest to them. I don't know for sure that 

9 an SSO is a broader concept than an SDO, but 

10 it could be. 

11 Q. What do you understand to be 

12 the constituents of the plaintiffs in this 

13 case? 

14 MR. FEE: Objection to form. 

15 THE WITNESS: I laid that out 

16 in my report. In summary, I believe 

17 they try to include in the process 

18 both those — both supply-side 

19 entities and demand-side entities. 

20 BY MR. BRIDGES: 

21 Q. Who else are plaintiffs' 

22 constituents? 

23 MR. FEE: Same objection. 

24 THE WITNESS: I can't think of 

25 anything that doesn't fall within 
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1 

Q. So those would be harms caused 

1 

think basically what I'm saying is 

2 

by a court decision? 

2 

what would - or addressing, is what 

3 

MR. FEE: Same objection. 

3 

would be the harm to the plaintiffs if 

4 

THE WITNESS: By continuing 

4 

there's no permanent injunction. 

5 

activities by the defendant that are 

5 

BY MR. BRIDGES: 

6 

not halted by the Court. 

6 

Q. Well, what did you mean by 

7 

BY MR. BRIDGES: 

7 

"losing copyright protection" in the 

8 

Q. Well, it comes across, frankly, 

8 

paragraph - in the heading VI on page 48? 

9 

in your report as though you're identifying 

9 

A. In essence, you can think of it 

10 

harms that would flow from a court decision. 

10 

as what would happen if there's no permanent 

11 

MR. FEE: Objection. 

11 

injunction. In other words, what the 

12 

BY MR. BRIDGES: 

12 

defendant has done in the past and what it's 

13 

Q. Is that correct or not? 

13 

likely to do in the future is allowed to 

14 

A. No, I think you — 

14 

continue. 

15 

MR. FEE: Mischaracterizes the 

15 

Q. And you immediately go into 

16 

report. 

16 

paragraph 112 talking about Emily Bremer, 

17 

THE WITNESS: - you misread 

17 

correct? 

18 

it. I don't think I said that or 

18 

A. I don't know what you mean by 

19 

meant to say that. 

19 

"immediately." It's the first paragraph in 

20 

BY MR. BRIDGES: 

20 

Section VI. 

21 

Q. So what harms have occurred 

21 

Q. Right. Was Emily Bremer in the 

22 

from the - from the defendant's conduct to 

22 

passage you referred to referring to the 

23 

date? 

23 

presence or absence of a permanent injunction 

24 

A. At the risk of repeating 

24 

in this case? 

25 

myself, some of that is summarized in 

25 

A. I don't think explicitly she 
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1 

paragraph 133, with regard to tangible 

1 

was addressing that issue, no. 

2 

evidence on harm. With regard to other 

2 

Q. Do you think implicitly she was 

3 

evidence, it's throughout the report. 

3 

referring to this case? 

4 

Q. So why would it make a 

4 

A. No. I thought you were asking 

5 

difference to what the defendant's harms 

5 

about permanent injunction. I don't think 

6 

are — strike - strike that. 

6 

she was addressing the — an injunction 

7 

Why would it make a defendants 

7 

issue. She was addressing the concept of 

8 

[sic] to the plaintiffs' harms if the 

8 

copyright protection. 

9 

plaintiffs' harms were continue with — 

9 

Q. And that's what you quoted her 

10 

strike that. 

10 

for, right, was for the concept of copyright 

11 

Is it your testimony that harms 

11 

protection for standards? 

12 

to plaintiffs would be different depending on 

12 

MR. FEE: Objection. You're 

13 

the particular basis of the Court's ruling? 

13 

referring just to paragraph 112? 

14 

MR. FEE: Objection. Vague. 

14 

BY MR. BRIDGES: 

15 

THE WITNESS: I-Idon't 

15 

Q. You may answer. 

16 

understand your question. 

16 

MR. FEE: Objection to form. 

17 

BY MR. BRIDGES: 

17 

THE WITNESS: I-Idon't 

18 

Q. It looks as though you're 

18 

understand the question. 

19 

stating what the harms would be if the Court 

19 

BY MR. BRIDGES: 

20 

found that incorporation by reference would 

20 

Q. You quoted her in 

21 

cause the plaintiffs to lose copyright 

21 

paragraph 112, correct? 

22 

protection; is that correct? 

22 

A. Yes. From one of her two 

23 

A. I don't - 

23 

articles, yes. 

24 

MR. FEE: Objection. Vague. 

24 

Q. Right. Regarding the concept 

25 

THE WITNESS: - think so. I 

25 

of copyright protection? 
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1 

A. Generally. I think she's 

1 

Q. "Such products" — 

2 

talking about standards development and 

2 

A. And in the next two sentences. 

3 

incorporation by reference. I don't remember 

3 

Q. And these are other products 

4 

if she said at the very beginning of the 

4 

that "could include more sophisticated 

5 

article that it was about copyright 

5 

Web-based availability, published 

6 

protection, but she certainly talks about 

6 

compilations of incorporated standards, and 

7 

copyright protection. 

7 

other ancillary products that incorporate the 

8 

Q. And you're quoting her about 

8 

standards"; isn't that correct? 

9 

losing copyright protection, and you're 

9 

A. You didn't read that right. It 

10 

placing it in the context of harms of the 

10 

starts "such products could include." 

11 

loss of copyright protection, correct? 

11 

Q. Okay. Otherwise, that reading 

12 

MR. FEE: Objection to form. 

12 

is correct, correct? 

13 

THE WITNESS: This excerpt 

13 

A. I think so. 

14 

doesn't specifically talk about losing 

14 

Q. You consider that to be harm to 

15 

copyright protection, but it talks 

15 

the plaintiffs? 

16 

about the concept of it. If there was 

16 

MR. FEE: Objection. Vague. 

17 

no longer copyright protection granted 

17 

THE WITNESS: It could be, yes. 

18 

to the SDOs, what would be the 

18 

It's likely to be, if the copyright 

19 

repercussions. 

19 

infringement or the assumption of a 

20 

BY MR. BRIDGES: 

20 

copyright infringement continues. It 

21 

Q. And that's the context that you 

21 

could broaden. 

22 

identified in the first line of 

22 

BY MR. BRIDGES: 

23 

paragraph 112, correct? 

23 

Q. Right. But the fact that these 

24 

A. Yes. 

24 

other types of products would enter the 

25 

MR. FEE: Objection to form. 

25 

marketplace is part of the harm that you 
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1 

BY MR. BRIDGES: 

1 

envision from the defendant in this case? 

2 

Q. Let me direct your attention to 

2 

MR. FEE: Objection to form. 

3 

paragraph 35 of your report. It says, "With 

3 

THE WITNESS: It's potential- 

4 

regard to expansion beyond the specific 

4 

there's a potential that the defendant 

5 

actions of Public Resource here, the 

5 

could do that. There's also the 

6 

'product' offerings of Public Resource - 

6 

potential that other parties could do 

7 

scans of paper copies of standards with some 

7 

that. 

8 

rekeying of text and some redrawing of 

8 

BY MR. BRIDGES: 

9 

diagrams (with some containing errors) - 

9 

Q. What - 

10 

represent a rudimentary first step in the use 

10 

A. I don't know for sure what the 

11 

of Plaintiffs' standards that is likely to 

11 

defendant has in mind. 

12 

become much more sophisticated if the Court 

12 

Q. Why did you take into account 

13 

holds that third parties are free to use 

13 

harms caused by other parties in this case? 

14 

Plaintiffs' standards with impunity after 

14 

A. Because — 

15 

they are incorporated by reference into law." 

15 

MR. FEE: Objection. Lack of 

16 

Do you see that? 

16 

foundation. 

17 

A. Yes, I do. 

17 

Go ahead. 

18 

Q. That is your statement, 

18 

THE WITNESS: If no copyright 

19 

correct? 

19 

protection is allowed here, in other 

20 

A. Yes. 

20 

words, there's no permanent 

21 

Q. What are the steps that you're 

21 

injunction, Public Resource and other 

22 

envisioning there beyond the rudimentary 

22 

parties like it will have freedom to 

23 

first step that you identify? 

23 

do what the plaintiffs believe they 

24 

A. I think they're laid out in the 

24 

should not have freedom to do. 

25 

next sentence. 

25 

BY MR. BRIDGES: 
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1 

Q. In other words, if the Court 

1 

standards. 

2 

makes a decision in a certain way, there will 

2 

Q. What further harm would 

3 

be harms from persons or entities other than 

3 

Public.Resource.Org cause to plaintiffs with 

4 

Public.Resource.Org to the plaintiffs? Is 

4 

respect to the standards at issue in this 

5 

that your testimony? 

5 

case if no - if the Court does not 

6 

MR. FEE: Objection to form. 

6 

permanently enjoin Public.Resource.Org? 

7 

THE WITNESS: You used the 

7 

MR. FEE: Objection to form. 

8 

phrase "in a certain way." I don't 

8 

THE WITNESS: If there's no 

9 

know what you mean by that. I'm 

9 

permanent injunction, there will, in 

10 

addressing the issue of whether there 

10 

essence, be a message sent to the 

11 

should be a permanent injunction or 

11 

marketplace that the standards that 

12 

not. 

12 

have already been disseminated are out 

13 

BY MR. BRIDGES: 

13 

there and can be used by others. 

14 

Q. So your view is that, if the 

14 

So right now my expectation is 

15 

Court does not enter a permanent injunction, 

15 

that some number of consumers of the 

16 

the plaintiffs will suffer harms from parties 

16 

standards have been reluctant or 

17 

other than Public.Resource.Org. Is that your 

17 

unknowing as to the standards 

18 

testimony? 

18 

disseminated by Public Resource. Now 

19 

A. That potential exists. I don't 

19 

there will be more knowledge about 

20 

know for sure. That's, in part, why the harm 

20 

that and more approval of that 

21 

is irreparable or very difficult to quantify. 

21 

activity. That is if there's no 

22 

Q. The - what harm? 

22 

permanent injunction. 

23 

A. Continuing activity of Public 

23 

BY MR. BRIDGES: 

24 

Resource and others. I don't know exactly 

24 

Q. What harms will plaintiffs 

25 

what will happen, but the potential is that 

25 

suffer if the Court rules that the plaintiffs 
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1 

there could be very broad dissemination of 

1 

do not own the copyrights in this case? 

2 

the standards, which would impact these SDOs 

2 

MR. FEE: Objection. Calls for 

3 

tremendously. 

3 

speculation. 

4 

Q. What harm would 

4 

THE WITNESS: In essence, 

5 

Public.Resource.Org cause to plaintiffs if 

5 

you're asking if there's no copyright 

6 

there is no permanent injunction? 

6 

infringement? 

7 

A. A permanent injunction would — 

7 

BY MR. BRIDGES: 

8 

lack of a permanent injunction would harm the 

8 

Q. No. What harms - have you 

9 

SDOs. 

9 

identified what harms the plaintiffs would 

10 

Q. That wasn't my question. My 

10 

suffer if the Court rules that the plaintiffs 

11 

question was, what harm would 

11 

do not own the copyrights at issue, that 

12 

Public.Resource.Org cause to plaintiffs if 

12 

there are no copyrights that the plaintiffs 

13 

there is no permanent injunction? 

13 

own — 

14 

A. At the very least, it's 

14 

MR. FEE: Objection to form. 

15 

associated with its historical dissemination 

15 

BY MR. BRIDGES: 

16 

of these standards, and there would be, in 

16 

Q. - at issue in this case? 

17 

essence, a carte blanche for other 

17 

A. I haven't addressed or thought 

18 

organizations or individuals to access those. 

18 

about that issue. There are also, don't 

19 

So my expectation is that the 

19 

forget, trademark issues. 

20 

dissemination of the materials that have 

20 

Q. I'm asking about copyright, so 

21 

already been disseminated will expand. 

21 

I ask you to confine your answers to my 

22 

It could also be the case that 

22 

questions. 

23 

Public Resource will undertake further 

23 

My question is, what — you 

24 

activities that would disseminate either 

24 

assume for purposes of your analysis that 

25 

already disseminated standards or other 

25 

plaintiffs own valid copyrights, correct? 
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1 

A. I assume that there's copyright 

1 

plaintiffs deserve copyright protection for 

2 

infringement. I don't know that I've made an 

2 

these standards? 

3 

explicit assumption with regard to ownership. 

3 

MR. FEE: Objection to form. 

4 

Q. And you assume infringement 

4 

THE WITNESS: I don't have an 

5 

without assuming ownership of the copyrights? 

5 

opinion on that one way or the other. 

6 

A. I haven't made any explicit 

6 

I have not thought about that topic. 

7 

assumption with regard to ownership. I know 

7 

BY MR. BRIDGES: 

8 

that's an issue in this case, but it's well 

8 

Q. Do you have any expertise in 

9 

beyond my expertise. 

9 

copyright law as a field of law? 

10 

Q. So if it turns out that — do 

10 

MR. FEE: Objection. Vague. 

11 

you understand your testimony to have any 

11 

THE WITNESS: No, I don't have 

12 

bearing on whether plaintiffs' standards are 

12 

legal expertise. I have expertise in 

13 

copyrightable? 

13 

looking at harm associated with 

14 

MR. FEE: Objection. Calls for 

14 

copyright infringement. 

15 

speculation. 

15 

BY MR. BRIDGES: 

16 

I would instruct you to not 

16 

Q. Do you have any expertise with 

17 

disclose any communications you had 

17 

respect to harm caused by invalidation of 

18 

with counsel that weren't the basis 

18 

copyrights? 

19 

for any of your opinions in this case. 

19 

MR. FEE: Same objection. 

20 

You can otherwise answer. 

20 

THE WITNESS: I'm not quite 

21 

THE WITNESS: Could you read 

21 

sure I'm fully appreciating your 

22 

that back or ask it again, please? 

22 

question. Again, I'm an expert in the 

23 

BY MR. BRIDGES: 

23 

economics of IP protection. One of 

24 

Q. Do you understand your 

24 

the areas in which I do work is harm 

25 

testimony and opinions in this case to have 

25 

associated with copyright protection. 


Page 78 


Page 80 

1 

any bearing on whether plaintiffs' standards 

1 

BY MR. BRIDGES: 

2 

are copyrightable? 

2 

Q. Have you done any work in this 

3 

MR. FEE: Same objection and 

3 

case to quantify what harms plaintiffs would 

4 

instruction. Plus objection, calls 

4 

suffer if a court were to rule that they 

5 

for a legal conclusion. 

5 

lacked copyright rights in the standards at 

6 

THE WITNESS: I don't know one 

6 

issue in this case? 

7 

way or the other. I've not taken on 

7 

MR. FEE: Objection to form. 

8 

that assignment. 

8 

Go ahead. 

9 

BY MR. BRIDGES: 

9 

THE WITNESS: Not explicitly, 

10 

Q. Do you understand whether your 

10 

to my knowledge. 

11 

testimony and opinions in this case are 

11 

BY MR. BRIDGES: 

12 

relevant to whether plaintiffs deserve 

12 

Q. Have you done anything 

13 

copyright protection in this case? 

13 

implicitly? 

14 

MR. FEE: Objection. Calls for 

14 

MR. FEE: Same objection. 

15 

a legal conclusion. 

15 

THE WITNESS: Not to my 

16 

And same objection with respect 

16 

knowledge. 

17 

to communications between you and 

17 

BY MR. BRIDGES: 

18 

counsel that were not the bases for 

18 

Q. Have you done any work in this 

19 

your opinions or your report. 

19 

case to analyze the incentives that 

20 

THE WITNESS: I don't know one 

20 

participants have in the standards 

21 

way or the other. I did not take on 

21 

development process? 

22 

that assignment. 

22 

MR. FEE: Objection to form. 

23 

BY MR. BRIDGES: 

23 

Vague. 

24 

Q. Do you mean by your analysis 

24 

THE WITNESS: I have in the 

25 

and opinions to suggest in any way that 

25 

sense that I've examined the materials 
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1 

Q. Right. Or approximately 

1 

THE WITNESS: Again, I don't 

2 

$3 million? 

2 

have an estimate. 

3 

A. Are you limiting it just to 

3 

BY MR. BRIDGES: 

4 

90.1 or all its standards? 

4 

Q. Do you know - did ASHRAE pay 

5 

Q. Well, that's a good question. 

5 

for the time, the hotel bills, and the plane 

6 

What — what's — what did you intend the 

6 

fares of its volunteer members in updating 

7 

last sentence in paragraph 76 to refer to? 

7 

the ASHRAE 90.1 standard? 

8 

All of its standards or 90.1? 

8 

A. I would expect rarely. It's 

9 

A. I think it's all of its 

9 

possible that there are certain instances in 

10 

standards, but we could visit the screenshot 

10 

which there was some set of out-of-pocket 

11 

from the Web site to confirm that. 

11 

expenses covered, but I would imagine the 

12 

Q. Okay. 

12 

bulk of the time it's the volunteer's 

13 

A. I — I could be wrong. I don't 

13 

employer. 

14 

think I am, but I could be. 

14 

MR. BRIDGES: Sorry. How long 

15 

Q. Okay. In the previous 

15 

have we been going? I didn't get when 

16 

sentence, you say, "ASHRAE and its volunteer 

16 

we went back on. 

17 

members devoted more than 86,400 man-hours, 

17 

MR. FEE: 34 minutes. 

18 

3,600 hotel nights, and 1,200 round-trip 

18 

BY MR. BRIDGES: 

19 

flights as part of the process." 

19 

Q. Did you speak with Emily Bremer 

20 

And that — "the process" 

20 

at any point in this case? 

21 

appears to refer to updating the ASHRAE 90.1 

21 

A. No. 

22 

standard, correct? 

22 

Q. How did you become acquainted 

23 

A. Yes. 

23 

with her writings? 

24 

Q. When you say "ASHRAE and its 

24 

A. I think Kevin Fee and/or 

25 

volunteer members," and then you give those 

25 

Jordana Rubel brought to my attention that 
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1 

statistics, those statistics refer primarily 

1 

she had written on this topic. I don't 

2 

to the man-hours, hotel nights, and 

2 

recall whether then we separately obtained 

3 

round-trip flights of the volunteer members? 

3 

her two articles or Mr. Fee slash Ms. Rubel 

4 

MR. FEE: Objection. Vague. 

4 

provided those to us. 

5 

THE WITNESS: Probably. As 

5 

Q. What independent work did you 

6 

opposed to ASHRAE-employed staff. 

6 

do to research writings regarding the 

7 

BY MR. BRIDGES: 

7 

economics of standards development? 

8 

Q. Do you know how much ASHRAE's 

8 

MR. FEE: Objection to form. 

9 

volunteer members and their employers — 

9 

THE WITNESS: We did 

10 

strike that. 

10 

independent research in the sense that 

11 

Do you know how much ASHRAE's 

11 

people that work with me did a 

12 

volunteer members and their employers spent 

12 

literature search to determine what 

13 

in salaries and disbursements for the 

13 

writings had been done in the area. 

14 

man-hours, hotel nights, and round-trip 

14 

I was previously aware of some 

15 

flights that were part of the process of 

15 

amount of the scholarship to begin 

16 

updating the ASHRAE 90.1 standard? 

16 

with. 

17 

A. I don't know, but it — I would 

17 

BY MR. BRIDGES: 

18 

imagine it's a noticeable amount, but I don't 

18 

Q. How is that literature search 

19 

know the amount. 

19 

reflected in any documents? 

20 

Q. What would be your best 

20 

A. The results are shown in my 

21 

estimate? 

21 

tab 2, and in particular it is page 2 of my 

22 

A. I don't have a best estimate. 

22 

tab 2, at the bottom. 

23 

Q. Would it be probably over 

23 

Q. And were these items found by 

24 

$10 million? 

24 

you or your team? 

25 

MR. FEE: Objection to form. 

25 

MR. FEE: Objection to form. 
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1 

THE WITNESS: Yes, with the 

1 

the various plaintiffs. 

2 

exception that, in the first instance, 

2 

Q. With whom? 

3 

lawyers at Morgan Lewis brought to our 

3 

A. They are all identified in 

4 

attention the Bremer — the existence 

4 

paragraph 10 of my report. 

5 

of Bremer articles. 

5 

Q. Which of those did you 

6 

BY MR. BRIDGES: 

6 

personally have conversations with? 

7 

Q. Did you study any of the 

7 

A. All of them, as I recall. It's 

8 

materials that Bremer — strike that. 

8 

possible there's someone I did not, but I'm 

9 

Bremer's articles are law 

9 

not remembering that being the case. 

10 

review articles, correct? 

10 

Q. Approximately how long did you 

11 

A. Yes. 

11 

spend with — did you have conversations with 

12 

Q. Did any plaintiff — did your 

12 

any of them together? 

13 

team's research identify any articles that 

13 

A. Yes, several of them were 

14 

you chose not to include in tab 2? 

14 

together. 

15 

A. I don't think so. 

15 

Q. Which ones? 

16 

Q. Did any plaintiff or its 

16 

A. I don't recall all 

17 

counsel furnish you with correspondence 

17 

combinations. I can say with some confidence 

18 

between the plaintiffs and Emily Bremer for 

18 

that there was never more than one plaintiff 

19 

review? 

19 

on a call. In other words, there were 

20 

A. No, not to my knowledge. 

20 

several people from a particular plaintiff on 

21 

Q. How many conversations with 

21 

a call, but not more than one plaintiff. 

22 

representatives of the plaintiffs did you 

22 

So I had various combinations 

23 

have? 

23 

of calls with ASTM that may have occurred on 

24 

MR. FEE: Objection. 

24 

three occasions; with NFPA, one or two 

25 

I would instruct you not to 

25 

occasions; and with ASHRAE, one or two 
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1 

answer questions regarding 

1 

occasions. 

2 

communications with counsel, unless 

2 

Q. And approximately how long 

3 

they formed the basis of your 

3 

total did you spend in conversations with 

4 

opinions, in which case you can answer 

4 

representatives of each plaintiff? 

5 

questions with respect to those 

5 

MR. FEE: Objection to form. 

6 

conversations. 

6 

THE WITNESS: Cumulatively, 

7 

BY MR. BRIDGES: 

7 

somewhere between three and five hours 

8 

Q. So I — I'll change my question 

8 

is my best guess right now. 

9 

slightly. 

9 

BY MR. BRIDGES: 

10 

How many — how many 

10 

Q. When you say cumulative - 

11 

conversations did you have with non-lawyer 

11 

"cumulatively," you mean for all plaintiffs? 

12 

employees or former employees of the 

12 

A. Yes. Meaning I'm - I've added 

13 

plaintiffs? 

13 

up the conversations I had across all three 

14 

A. None that the — that did not 

14 

plaintiffs. 

15 

include the lawyers. 

15 

Q. Right. What's your best 

16 

Q. Right. I'm — so I'm asking 

16 

estimate as to the period of time you spent 

17 

you to tell me what they were. If the 

17 

with each plaintiff? 

18 

presence of lawyer - if you had a 

18 

A. With ASTM, it may have been two 

19 

conversation with a — with an employee or 

19 

to three hours. For NFPA, one to two hours. 

20 

former employee of the plaintiff, I'd like to 

20 

For ASHRAE, one to two hours. That's my best 

21 

know what that was. So the fact that lawyers 

21 

guess right now. 

22 

may have been present wouldn't excuse it from 

22 

* * * 

23 

the scope of the answer. 

23 

(Jarosz Exhibit 2 and Jarosz-3 

24 

A. I had somewhere between four 

24 

marked for identification.) 

25 

and six conversations with people who were at 

25 

* * * 
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1 

record at 12:17. This is the end of 

1 

you interview? 

2 

media unit number 1. 

2 

A. I don't think I interviewed any 

3 

* * * 

3 

members of the public either. 

4 

(Recess from 12:17 p.m. to 

4 

Q. What steps did you do to 

5 

12:32 p.m.) 

5 

ascertain the views of the members of the 

6 

* * * 

6 

organizations, other than the employees? 

7 

THE VIDEOGRAPHER: On the 

7 

A. I read the materials that were 

8 

record at 12:32. This is the 

8 

produced here. I read the deposition 

9 

beginning of media unit 2 in the 

9 

testimony of the various individuals. I read 

10 

deposition of John Jarosz. 

10 

the articles published by Ms. Bremer. And I 

11 

BY MR. BRIDGES: 

11 

read the other academic literature and 

12 

Q. Mr. Jarosz, your report, as I 

12 

practical literature that I had. 

13 

referred to earlier, cites a number of 

13 

Q. Which of those sources stated 

14 

conversations with employees of the 

14 

the views of the non-employee members of the 

15 

plaintiffs. For what purpose did you have 

15 

various organizations? 

16 

conversations with the plaintiffs' employees? 

16 

A. I don't know that views of — 

17 

A. To learn more about the 

17 

that their views were explicitly addressed in 

18 

organization and their view as to the impact 

18 

my report or represented. I understood what 

19 

of continued copyright protection — 

19 

the impacts of the lack of honoring the 

20 

continued copyright infringement and 

20 

copyrights and trademarks would have, but I 

21 

trademark infringement. 

21 

don't know that I saw non-employee member 

22 

Q. What view did you learn from 

22 

views explicitly summarized. 

23 

them? 

23 

Q. So what steps did you do to 

24 

MR. FEE: Objection to form. 

24 

ascertain the views of the members of the 

25 

THE WITNESS: Well, I solicited 

25 

organizations — 
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1 

and learned many facts about the 

1 

MR. FEE: Objection. 

2 

organizations. I also learned that 

2 

BY MR. BRIDGES: 

3 

each one of them viewed continued 

3 

Q. - other than their employees? 

4 

copyright infringement and trademark 

4 

MR. FEE: Asked and answered. 

5 

infringement as quite detrimental to 

5 

THE WITNESS: Well, I talked to 

6 

their organizations, detrimental to 

6 

the employees, and they interact with 

7 

the members, detrimental to the 

7 

the members on a very regular basis, 

8 

public. 

8 

so they gave me some sense of what the 

9 

They viewed continued IP 

9 

views of the members were. 

10 

infringement as potentially 

10 

It also could be that some of 

11 

devastating to their organizations. 

11 

the perspectives of the members are 

12 

BY MR. BRIDGES: 

12 

reflected in some of the documents I 

13 

Q. These were their views? 

13 

identified in tab 2. 

14 

A. Yes. I'm just paraphrasing, of 

14 

BY MR. BRIDGES: 

15 

course. 

15 

Q. Well, I'm just trying to find 

16 

Q. What members did you interview? 

16 

out where — it sounds as though - strike 

17 

A. None, other than the employees. 

17 

that. 

18 

I don't know if you call those "members" or 

18 

It sounds as though a minute 

19 

not. But the volunteer membership, I didn't 

19 

ago you said you couldn't recall anything 

20 

goto. 

20 

specifically calling out views of 

21 

THE VIDEOGRAPHER: Excuse me. 

21 

non-employee members, correct? 

22 

Counsel, could you move your 

22 

A. Correct. I think that's right. 

23 

microphone to your lapel? Thank you. 

23 

Q. What did you do to verify the 

24 

BY MR. BRIDGES: 

24 

statements that employees of the plaintiffs 

25 

Q. What members of the public did 

25 

made about the views of the non-employee 
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1 

members of their organizations? 

1 

States other than law review articles by 

2 

A. I did what I normally do in an 

2 

Emily Bremer? 

3 

assignment like this and look at the produced 

3 

A. As I sit here right now, I'm 

4 

materials. 

4 

not aware of any documents that discuss the 

5 

Q. And the produced materials did 

5 

deliberations, but my memory is not perfect. 

6 

not call out specifically any views of 

6 

Q. Do you know if there was a 

7 

non-employee members of the plaintiff 

7 

consensus in any relevant committee of the 

8 

organizations, correct? 

8 

Administrative Conference of the United 

9 

A. I don't recall any specific 

9 

States regarding the conclusions that 

10 

views being summarized. My memory may not be 

10 

Ms. Bremer states in her law review articles? 

11 

perfect on that, though. 

11 

A. I don't. 

12 

Q. What research, if any, did you 

12 

MR. FEE: Objection. Vague. 

13 

do among members of the public about whether 

13 

BY MR. BRIDGES: 

14 

lack of copyright protection for the 

14 

Q. Do you know whether there was 

15 

plaintiffs' standards would be detrimental to 

15 

any dissent in any relevant committee of the 

16 

the — to the public? 

16 

Administrative Conference of the United 

17 

A. The information that I reviewed 

17 

States regarding the conclusions that 

18 

is in tab 2. I didn't have material beyond 

18 

Ms. Bremer states in her law review articles? 

19 

what is identified in tab 2. 

19 

MR. FEE: Objection to form. 

20 

Q. So what in tab 2 reflects your 

20 

THE WITNESS: I don't. 

21 

steps to ascertain the views of members of 

21 

BY MR. BRIDGES: 

22 

the public? 

22 

Q. Do you know why persons get 

23 

MR. FEE: Objection to form. 

23 

appointed to the Administrative Conference of 

24 

THE WITNESS: I think the 

24 

the United States? 

25 

Bremer articles, in part, address 

25 

A. I may have known that, but I 
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1 

that. I think some of the federal 

1 

don't recall that sitting here now. 

2 

government's circulars that I 

2 

Q. Do you know whether 

3 

identify, in part, reflect the 

3 

Ms. Bremer's articles — strike that. 

4 

reviews, in particular the NTTAA of 

4 

Do you know whether 

5 

1995 and OMB Circular A-l 19. I think 

5 

Ms. Bremer's law review articles reflect a 

6 

they, in part, reflect public views. 

6 

view of the Administrative Conference of the 

7 

There are probably other things. 

7 

United States - 

8 

BY MR. BRIDGES: 

8 

MR. FEE: Objection to form. 

9 

Q. Did you review OMB Circular 

9 

BY MR. BRIDGES: 

10 

A-l 19 personally? 

10 

Q. - or of any of its committees? 

11 

A. Yes. As I recall, I did. 

11 

MR. FEE: Objection to form. 

12 

Q. Did you review any materials 

12 

THE WITNESS: I'm not aware 

13 

pertaining to the discussions or 

13 

that they officially reflect that. I 

14 

deliberations of the Administrative 

14 

believe she gathered information, and 

15 

Conference of the United States in connection 

15 

they may, in fact, represent the views 

16 

with your research or analysis? 

16 

of some or all members, but I don't 

17 

A. What particular materials or 

17 

think that's - that either article is 

18 

meetings are you referring to? 

18 

an official representation - 

19 

Q. Any. 

19 

BY MR. BRIDGES: 

20 

A. I don't recall, but it's 

20 

Q. Are you — 

21 

possible. 

21 

A. - of that body. 

22 

Q. Does tab 2 refer you to any 

22 

Q. Are you aware of the fact that 

23 

documents that would provide you information 

23 

her articles — her law review articles 

24 

about the discussions or deliberations of the 

24 

specifically disclaim her articles as the 

25 

Administrative Conference of the United 

25 

views of any government entity and indicate 
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1 

members of their organizations? 

1 

States other than law review articles by 

2 

A. I did what I normally do in an 

2 

Emily Bremer? 

3 

assignment like this and look at the produced 

3 

A. As I sit here right now, I'm 

4 

materials. 

4 

not aware of any documents that discuss the 

5 

Q. And the produced materials did 

5 

deliberations, but my memory is not perfect. 

6 

not call out specifically any views of 

6 

Q. Do you know if there was a 

7 

non-employee members of the plaintiff 

7 

consensus in any relevant committee of the 

8 

organizations, correct? 

8 

Administrative Conference of the United 

9 

A. I don't recall any specific 

9 

States regarding the conclusions that 

10 

views being summarized. My memory may not be 

10 

Ms. Bremer states in her law review articles? 

11 

perfect on that, though. 

11 

A. I don't. 

12 

Q. What research, if any, did you 

12 

MR. FEE: Objection. Vague. 

13 

do among members of the public about whether 

13 

BY MR. BRIDGES: 

14 

lack of copyright protection for the 

14 

Q. Do you know whether there was 

15 

plaintiffs' standards would be detrimental to 

15 

any dissent in any relevant committee of the 

16 

the — to the public? 

16 

Administrative Conference of the United 

17 

A. The information that I reviewed 

17 

States regarding the conclusions that 

18 

is in tab 2. I didn't have material beyond 

18 

Ms. Bremer states in her law review articles? 

19 

what is identified in tab 2. 

19 

MR. FEE: Objection to form. 

20 

Q. So what in tab 2 reflects your 

20 

THE WITNESS: I don't. 

21 

steps to ascertain the views of members of 

21 

BY MR. BRIDGES: 

22 

the public? 

22 

Q. Do you know why persons get 

23 

MR. FEE: Objection to form. 

23 

appointed to the Administrative Conference of 

24 

THE WITNESS: I think the 

24 

the United States? 

25 

Bremer articles, in part, address 

25 

A. I may have known that, but I 
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1 

that. I think some of the federal 

1 

don't recall that sitting here now. 

2 

government's circulars that I 

2 

Q. Do you know whether 

3 

identify, in part, reflect the 

3 

Ms. Bremer's articles — strike that. 

4 

reviews, in particular the NTTAA of 

4 

Do you know whether 

5 

1995 and OMB Circular A-l 19. I think 

5 

Ms. Bremer's law review articles reflect a 

6 

they, in part, reflect public views. 

6 

view of the Administrative Conference of the 

7 

There are probably other things. 

7 

United States — 

8 

BY MR. BRIDGES: 

8 

MR. FEE: Objection to form. 

9 

Q. Did you review OMB Circular 

9 

BY MR. BRIDGES: 

10 

A-l 19 personally? 

10 

Q. — or of any of its committees? 

11 

A. Yes. As I recall, I did. 

11 

MR. FEE: Objection to form. 

12 

Q. Did you review any materials 

12 

THE WITNESS: I'm not aware 

13 

pertaining to the discussions or 

13 

that they officially reflect that. I 

14 

deliberations of the Administrative 

14 

believe she gathered information, and 

15 

Conference of the United States in connection 

15 

they may, in fact, represent the views 

16 

with your research or analysis? 

16 

of some or all members, but I don't 

17 

A. What particular materials or 

17 

think that's — that either article is 

18 

meetings are you referring to? 

18 

an official representation — 

19 

Q. Any. 

19 

BY MR. BRIDGES: 

20 

A. I don't recall, but it's 

20 

Q. Are you — 

21 

possible. 

21 

A. — of that body. 

22 

Q. Does tab 2 refer you to any 

22 

Q. Are you aware of the fact that 

23 

documents that would provide you information 

23 

her articles — her law review articles 

24 

about the discussions or deliberations of the 

24 

specifically disclaim her articles as the 

25 

Administrative Conference of the United 

25 

views of any government entity and indicate 
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1 

that they are her personal views? 

1 

I believe counsel did not 

2 

A. I wouldn't be surprised and 

2 

provide the Web site screenshots, but I might 

3 

may — I may have read that, but I would 

3 

be wrong on that. 

4 

expect that that would be in the first 

4 

Q. And did you do anything — 

5 

footnote of one or both articles. 

5 

what, if anything, did you do to test the 

6 

Q. What did you do to examine the 

6 

validity of the factual assertions that the 

7 

alleged facts that the representatives of 

7 

plaintiffs made to you in your conversations 

8 

plaintiffs stated to you in their 

8 

with their employees? 

9 

conversations with you? 

9 

MR. FEE: Objection to form. 

10 

MR. FEE: Objection to form. 

10 

Asked and answered. 

11 

THE WITNESS: I looked at- 

11 

THE WITNESS: Well, we looked 

12 

MR. FEE: Asked and answered. 

12 

at materials. If we found things that 

13 

THE WITNESS: I'm sorry. I 

13 

conflicted with what we learned, that 

14 

looked at the document production and 

14 

would prompt us to investigate 

15 

the other materials shown in tab 2. 

15 

further. But I don't recall seeing 

16 

BY MR. BRIDGES: 

16 

any documentary evidence that 

17 

Q. You looked at the document 

17 

conflicted with facts that were 

18 

production that the plaintiffs' counsel 

18 

provided by plaintiff personnel, but I 

19 

furnished you? 

19 

might be wrong. 

20 

A. In part. There were other 

20 

BY MR. BRIDGES: 

21 

things in tab 2 that were not provided to me 

21 

Q. Did you investigate 

22 

by plaintiffs' counsel. 

22 

independently whether documents existed that 

23 

Q. What other materials in 

23 

contradicted plaintiffs' statements of facts? 

24 

tab 2 — strike that. 

24 

A. Not with that in mind. We 

25 

Please identify for me in tab 2 

25 

looked at the documents and were mindful of 
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1 

the materials that plaintiffs' counsel 

1 

whether there were conflicts within documents 

2 

furnished you. 

2 

or conflicts between documents and other 

3 

A. I don't know with absolute 

3 

information, but I don't recall that we saw 

4 

certainty, but let me give you my best guess. 

4 

anything that gave us substantial pause. 

5 

I believe all the depositions that are shown 

5 

There were probably some things 

6 

on page 1. I believe the Bates ranges at the 

6 

where there were some uncertainties whether 

7 

very top of the page were provided by 

7 

there was a conflict or not and some where 

8 

counsel. 

8 

there were insignificant conflicts, but I 

9 

The deposition transcripts and 

9 

think mostly the information we saw did not 

10 

exhibits were provided by counsel. I believe 

10 

conflict with the information we learned from 

11 

the financial statements and plans were 

11 

plaintiff personnel. 

12 

provided by counsel. I believe the legal 

12 

Q. Did you investigate 

13 

documents were provided by counsel. I 

13 

independently whether other documents, apart 

14 

believe the miscellaneous items were provided 

14 

from the documents plaintiffs furnished you, 

15 

by counsel. 

15 

existed that contradicted plaintiffs' 

16 

I don't know about the cases 

16 

statements of facts — 

17 

and laws. I just don't remember if we 

17 

MR. FEE: Objection to form. 

18 

separately gathered those or were provided 

18 

BY MR. BRIDGES: 

19 

those. 

19 

Q. — in conversations with you? 

20 

The analyst reports, articles, 

20 

A. Yes, in the sense that we 

21 

books, and presentations, I think we gathered 

21 

gathered some information that we did not 

22 

all of those, with the possible exception of 

22 

receive from plaintiffs' counsel, but all of 

23 

the two Bremer articles. I don't recall if 

23 

that is identified in tab 2. 

24 

counsel provided that or we obtained those 

24 

Q. Which part of tab 2? 

25 

separately. 

25 

A. Well, as I said, I think the 
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1 

Web sites we gathered ourselves, and I think 

1 

Q. What did you hear about 

2 

the reports and articles, with the exception 

2 

overseas litigation involving Public 

3 

of the Bremer articles, we gathered 

3 

Resource? 

4 

ourselves. 

4 

A. I think I heard that there was 

5 

Q. Do you know why you got no 

5 

a German — or a suit in Germany, but I'm not 

6 

documents from NFPA, no Bates range documents 

6 

sure that I learned much more than that. I 

7 

from NFPA? 

7 

don't recall what status that suit - what 

8 

MR. REHN: Object to form - 

8 

the status of that suit is. 

9 

THE WITNESS: I don't know why 

9 

Q. Do you recall anyone disclosing 

10 

we did not receive Bates documents — 

10 

to you litigation involving NFPA in the 

11 

THE REPORTER: Wait. 

11 

United States that pertained to standards and 

12 

MR. REHN: Sorry. Object to 

12 

copyright? 

13 

the form. Lacks foundation. 

13 

A. It's possible, but I don't 

14 

THE WITNESS: I don't know for 

14 

recall any, sitting here right now. 

15 

sure that we didn't receive 

15 

Q. Do you recall inquiring about 

16 

Bates-stamped documents, but I believe 

16 

public statements of fact that NFPA has made 

17 

some of the documents we received were 

17 

regarding copyright and standards in 

18 

NFPA documents. 

18 

litigation other than this litigation in the 

19 

BY MR. BRIDGES: 

19 

United States? 

20 

Q. Do you recall seeing any NFPA 

20 

MR. FEE: Objection to form. 

21 

documents that — in which NFPA personnel 

21 

THE WITNESS: I do not. 

22 

stated that they could not show any harm from 

22 

BY MR. BRIDGES: 

23 

the defendant's activities? 

23 

Q. Are you familiar with a case 

24 

A. Received any documents that 

24 

called Veeck, V-E-E-C-K? 

25 

said that? 

25 

A. I'm familiar with an opinion in 
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1 

Q. Uh-huh. 

1 

the Veeck case. 

2 

A. Perhaps you would have 

2 

Q. What do you know about that 

3 

something that would refresh my memory. I 

3 

opinion? 

4 

don't recall, sitting here right now, but 

4 

MR. FEE: Objection. 

5 

it's possible. 

5 

I would instruct you not to 

6 

Are you talking about 

6 

disclose anything you know about that 

7 

historical — historically no harm, or are 

7 

opinion that was a result of 

8 

you talking about prospectively? 

8 

communications with counsel and that 

9 

Q. Either one. Did you — do you 

9 

did not form the basis of any of the 

10 

recall seeing any internal NFPA documents 

10 

opinions in your report or any of the 

11 

that call into question where NF — whether 

11 

assumptions that you relied upon in 

12 

NFPA has suffered any harm from the 

12 

reaching your conclusions. 

13 

defendant's activities? 

13 

THE WITNESS: I did talk with 

14 

A. I don't recall documents on it. 

14 

counsel about that case, and that case 

15 

There may have been some deposition testimony 

15 

didn't form any basis for any of my 

16 

about past activities, but I don't know if it 

16 

observations or conclusions here. 

17 

was activities prior to Public Resource 

17 

BY MR. BRIDGES: 

18 

actions here or after. 

18 

Q. Why did the Veeck case not form 

19 

Q. Do you recall learning about 

19 

any basis for any of your observations or 

20 

any litigation that NFPA had engaged in 

20 

conclusions here? 

21 

pertaining to standards and copyright? 

21 

A. I don't know how to answer that 

22 

A. I think I heard that there's 

22 

question. I — it didn't present any facts 

23 

some overseas litigation involving Public 

23 

that were specific to this case, as far as I 

24 

Resource. Whether that involves NFPA, I 

24 

recall. 

25 

don't know. 

25 

Q. What do you recall of the facts 
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1 

answered. 

1 

A. Not sitting here right now, I 

2 

THE WITNESS: Again, I read the 

2 

don't. 

3 

case. I didn't do any analysis beyond 

3 

Q. Do you know whether ASHRAE took 

4 

that of that particular case. 

4 

over development of what became standard 90.1 

5 

BY MR. BRIDGES: 

5 

from any other group or entity? 

6 

Q. What steps did you take to 

6 

A. No, I do not. 

7 

ascertain what public harms flowed from the 

7 

Q. Have you ever quantified the 

8 

Court's decision in the Veeck case? 

8 

value of the contributions made by the 

9 

A. Other than reading the case, 

9 

volunteers of the various organizations to 

10 

the opinion in the case, I didn't do anything 

10 

the standards at issue in this case? 

11 

beyond that to understand the implications of 

11 

MR. FEE: Objection to form. 

12 

that holding. 

12 

THE WITNESS: Not other than 

13 

Q. You didn't do any investigation 

13 

having some sense of hours or a 

14 

as to the economic consequences to any 

14 

limited sense of dollars, hut not 

15 

entity, industry, or person as a consequence 

15 

beyond that, no. 

16 

of the decision in the Veeck case, correct? 

16 

BY MR. BRIDGES: 

17 

MR. FEE: Objection to form. 

17 

Q. Can you put a rough dollar 

18 

THE WITNESS: I think that's 

18 

value on the time and expenses of the 

19 

correct, yes. 

19 

volunteers with respect to any of the 

20 

BY MR. BRIDGES: 

20 

standards in this case? 

21 

Q. How has the process of 

21 

MR. FEE: Objection to form. 

22 

standards development changed in the last 100 

22 

THE WITNESS: Not sitting here 

23 

years, to your knowledge? 

23 

right now. That would entail a little 

24 

A. I don't know the specifics, and 

24 

hit of a study. I have not done that. 

25 

I don't know that there is one standards 

25 

BY MR. BRIDGES: 
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1 

development process. I think there are a 

1 

Q. What — what would be required? 

2 

variety of processes pursued by a number of 

2 

A. To understand basically the 

3 

SSOs or SDOs. I'm sure that there have been 

3 

out-of-pocket expenses incurred and the 

4 

changes on the margin. There may have been 

4 

opportunity costs incurred. So among other 

5 

larger changes. I just don't know. I have 

5 

things, one would want to look at time 

6 

not studied the trend in the standard 

6 

records, have an understanding of 

7 

development process over time. 

7 

compensation, have an understanding of the 

8 

Q. What changes are you aware of 

8 

activities of those individuals. Those 

9 

in the standards development process of NFPA 

9 

are — would be among the inputs. 

10 

over the past 100 years? 

10 

Q. What changes are you aware of 

11 

A. I don't know. I've not studied 

11 

in the distribution of standards in the past 

12 

that topic. 

12 

100 years by the plaintiffs? 

13 

Q. What changes are you aware of 

13 

MR. FEE: Objection to form. 

14 

in the standards development process of the 

14 

THE WITNESS: I haven't 

15 

ASHRAE 90.1 standard? 

15 

investigated that particular issue, 

16 

A. I don't know. I've not studied 

16 

but I understand that some of the 

17 

that. 

17 

standards today are distributed 

18 

Q. How did ASHRAE come to develop 

18 

through the Internet that certainly 

19 

the 90.1 standard? 

19 

didn't exist 100 years ago. 

20 

A. I think, generally, a need was 

20 

Some of the standards are 

21 

identified and a group of constituents 

21 

distributed for free with limitations. 

22 

convened to derive a standard, but I don't 

22 

I don't know if that was true 100 

23 

know the specifics beyond that. 

23 

years ago, but it might have been. 

24 

Q. Do you know who identified the 

24 

I would expect some of the 

25 

need? 

25 

copying and dissemination capabilities 
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1 

are much greater today than they were 

1 

the right to reproduce, copy, or 

2 

in 1915, but I don't know that the 

2 

disseminate those standards but can 

3 

general methods of — I don't know how 

3 

look at them online. 

4 

the general methods of distribution 

4 

BY MR. BRIDGES: 

5 

have changed. 

5 

Q. Have you used the reading rooms 

6 

BY MR. BRIDGES: 

6 

of any of the plaintiffs? 

7 

Q. What changes are you aware of 

7 

A. No, I have not. 

8 

in sales trends over the past 20 years? 

8 

Q. Have you reviewed the interface 

9 

MR. FEE: Objection to form. 

9 

that the - have you reviewed the interfaces 

10 

THE WITNESS: I don't have data 

10 

that the plaintiffs offer to persons wishing 

11 

going back as far as 20 years ago. I 

11 

to view materials for free online? 

12 

have some information on publication 

12 

A. No, I don't think so. 

13 

sales, for instance, in tabs 3,4, and 

13 

Q. Do you know what effect, if 

14 

5. They only — that information only 

14 

any, the presence of those free materials on 

15 

goes back a few years, however. 

15 

the plaintiffs' Web sites has had on the 

16 

BY MR. BRIDGES: 

16 

plaintiffs' revenues? 

17 

Q. Did you review any information 

17 

MR. FEE: Objection to form. 

18 

earlier than the dates shown in the documents 

18 

THE WITNESS: No, I don't. 

19 

at tabs 3, 4, and 5? 

19 

BY MR. BRIDGES: 

20 

MR. FEE: Objection. Vague. 

20 

Q. Have you — have you 

21 

THE WITNESS: It's possible 

21 

investigated that? 

22 

that some of the source documents had 

22 

MR. FEE: Same objection. 

23 

earlier information, but I don't 

23 

THE WITNESS: I've been 

24 

recall that. I would need to look at 

24 

opening — I've been open to learning 

25 

those source documents. 

25 

about that, but I haven't learned that 


Page 134 


Page 136 

1 

BY MR. BRIDGES: 

1 

there's a direct or indirect effect. 

2 

Q. And those source documents 

2 

There might be, but I haven't seen 

3 

would be within the Bates ranges identified 

3 

evidence of that. 

4 

in tab 2 of your report? 

4 

BY MR. BRIDGES: 

5 

A. Within the Bates ranges or 

5 

Q. My question was, have you 

6 

identified elsewhere in tab 2. For instance, 

6 

investigated that? 

7 

the AS team — ASTM audited - audited 

7 

MR. FEE: Same objection. 

8 

consolidated financial statements, I think, 

8 

THE WITNESS: Perhaps you could 

9 

may not all be Bates-stamped. I could be 

9 

read back my answer. 

10 

wrong on that. But I would look in that set 

10 

BY MR. BRIDGES: 

11 

of financial documents. 

11 

Q. I've heard the answer. It was 

12 

Q. What do you know about what you 

12 

not responsive to my question. The - you 

13 

said — strike that. 

13 

said you did not know what effect, if any, 

14 

You said earlier that some 

14 

the presence of those free materials on the 

15 

standards are distributed for free with some 

15 

plaintiffs' Web sites has had on the 

16 

limitations; is that correct? 

16 

plaintiffs' revenues. 

17 

A. Yes, that's my understanding. 

17 

And my question is, have you 

18 

Q. What do you know about that? 

18 

investigated that? 

19 

MR. FEE: Objection. Vague. 

19 

MR. FEE: Same objection. 

20 

THE WITNESS: I've written 

20 

THE WITNESS: No, I've not 

21 

about that in my report. I believe 

21 

undertaken a separate investigation. 

22 

that each one of the plaintiffs has 

22 

I've been alert to that topic, but I 

23 

provided what is sometimes called a 

23 

haven't assigned myself that 

24 

"reading room" so that people can look 

24 

investigation. 

25 

at those standards but are not given 

25 

BY MR. BRIDGES: 
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1 

Q. Was something that was — 

1 

SDOs, but the standard setting organizations 

2 

remained pending at the time you wrote this 

2 

that are the candidates are the ones that I 

3 

report as something that you expected to do 

3 

identified earlier today. 

4 

in the future? 

4 

Q. Which SDOs do you recall 

5 

A. No. 

5 

treating copyright protection of their 

6 

MR. FEE: Objection. Vague. 

6 

standards as very important? 

7 

THE WITNESS: I'm sorry. 

7 

A. I just don't recall right now. 

8 

No. 

8 

I — I have some vague recollection that 

9 

BY MR. BRIDGES: 

9 

copyright considerations are addressed by 

10 

Q. Did you study the practices of 

10 

ETSI, but I could be wrong on that. 

11 

any standards development organizations, 

11 

Q. What do you know about policies 

12 

other than the plaintiffs, for purposes of 

12 

or practices of the Blu-ray organization with 

13 

your work in this case? 

13 

respect to copyright protection? 

14 

MR. FEE: Objection. Vague. 

14 

A. I assume you're talking about 

15 

THE WITNESS: Not that I 

15 

the Blu-ray Association? I may have known 

16 

recall. I saw reference to other SDOs 

16 

when I was involved in that matter. I do not 

17 

in the Bremer articles, for instance, 

17 

remember, sitting here now. 

18 

but I didn't undertake a separate 

18 

Q. Do you recall that your report 

19 

investigation of the practices of any 

19 

actually refers to the Blu-ray Association? 

20 

other SDOs for purposes of my 

20 

A. I think I refer to Blu-ray 

21 

assignment here. 

21 

standards. I don't recall if I refer to the 

22 

BY MR. BRIDGES: 

22 

Blu-ray Association, but perhaps you could 

23 

Q. Are you aware of practices or 

23 

refresh my memory. 

24 

policies of other SDOs with reference to 

24 

Q. I believe you point it out at 

25 

either copyright or free availability of 

25 

the bottom of page 62. "While certain SDOs 
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1 

their materials? 

1 

(e.g., the Blu-ray disc association) provide 

2 

MR. FEE: Objection to form. 

2 

unrestricted access to their standard 

3 

THE WITNESS: I may have been 

3 

publications for free, the Plaintiffs here do 

4 

aware through other assignments I've 

4 

not." 

5 

undertaken in the past, but I didn't 

5 

Do you recall that? 

6 

undertake any separate investigation 

6 

A. Now I do. Thank you for 

7 

for purposes of this matter. 

7 

refreshing my memory. 

8 

BY MR. BRIDGES: 

8 

Q. What economic effects are you 

9 

Q. What awareness do you have of 

9 

aware of the fact that the Blu-ray Disc 

10 

the practices or policies of other SDOs 

10 

Association provides unrestricted access to 

11 

through other assignments you've undertaken 

11 

its standard publications for free? 

12 

in the past? 

12 

A. I have not investigated that 

13 

MR. FEE: Objection to form. 

13 

issue, so I don't know. 

14 

THE WITNESS: I can only recall 

14 

Q. What other SDOs have you 

15 

most generally that they view 

15 

identified that provide unrestricted access 

16 

intellectual property protection as 

16 

to their standards for free? 

17 

being very important, but I can't be 

17 

A. I don't think I've identified 

18 

any more specific than that. 

18 

any others in my report. 

19 

BY MR. BRIDGES: 

19 

Q. Did you look for any others? 

20 

Q. Which SDOs you — do you recall 

20 

A. Not that I recall. 

21 

treating intellectual property protection as 

21 

Q. Why not? 

22 

very important? 

22 

A. I don't know how to answer 

23 

A. Well, again, I've — I've dealt 

23 

that. I was aware of the Blu-ray Disc 

24 

with standards setting organizations. I 

24 

Association's policy in this regard, so I 

25 

don't know if any of those are technically 

25 

wrote about it here. 
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1 

Q. Why did you not consider the 

1 

perspective. 

2 

economic effects of free distribution of 

2 

BY MR. BRIDGES: 

3 

standards with respect to other 

3 

Q. And what is the relevance of 

4 

organizations? 

4 

economic analysis to that question, as you 

5 

A. I didn't quite see the 

5 

understand it? 

6 

relevance to this matter. 

6 

MR. FEE: Objection to form. 

7 

Q. Why? 

7 

Vague. Might also be construed to 

8 

A. I don't know how to prove a 

8 

require a legal conclusion. 

9 

negative. 

9 

THE WITNESS: Economists have a 

10 

Q. What's the negative you were 

10 

view and perspective at looking at 

11 

thinking of that would need to be proved or 

11 

issues that some courts have found to 

12 

disproved? 

12 

be useful. 

13 

A. That something is not relevant. 

13 

BY MR. BRIDGES: 

14 

Q. You just didn't see the 

14 

Q. Well, I'm asking, with specific 

15 

relevance? 

15 

relevance to this case, what do you 

16 

A. I don't understand how that 

16 

understand the importance of economic 

17 

would be helpful in the assignment that I had 

17 

analysis to be in this case — 

18 

here. 

18 

MR. FEE: Objection. Calls - 

19 

Q. And what was the assignment you 

19 

BY MR. BRIDGES: 

20 

had here? 

20 

Q. — as you have purported to 

21 

A. Well, I've laid it out — 

21 

practice it? 

22 

Q. I can read the report. I'm not 

22 

MR. FEE: Calls for a legal 

23 

asking you to read — read the report. I'd 

23 

conclusion. 

24 

like your own words now, sitting here. 

24 

Also, to the extent that 

25 

MR. FEE: Objection. 

25 

responding to that would require you 
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1 

BY MR. BRIDGES: 

1 

to disclose communications with 

2 

Q. How do you — how do you 

2 

counsel that did not form the basis 

3 

view — 

3 

for any of your opinions or 

4 

A. I'd like to answer it by 

4 

conclusions and did not provide any 

5 

looking at my report. 

5 

assumptions that were the basis for 

6 

Q. No, I'd like for you to give me 

6 

your opinions or conclusions, you 

7 

a straight answer, because if you're just 

7 

should not answer that portion of the 

8 

going to refer to the report, the report will 

8 

question. 

9 

speak for itself, and I don't need you to 

9 

THE WITNESS: I understand 

10 

read it to me. 

10 

that, generally, economists like me 

11 

I'd like for you to tell me 

11 

are quite helpful in determining 

12 

what you understand, sitting here, to have 

12 

questions of harm, particularly harm 

13 

been your assignment in this case. 

13 

as it relates to infringement of IP 

14 

MR. FEE: Objection. 

14 

rights. 

15 

You can answer the question 

15 

BY MR. BRIDGES: 

16 

however you deem appropriate. 

16 

Q. How do you distinguish between 

17 

THE WITNESS: I've aptly laid 

17 

harms that are caused by an infringement by 

18 

it out in my report, so I defer to the 

18 

the defendant versus harms that might be 

19 

words in my report. 

19 

caused by a court decision that plaintiffs 

20 

But I've, in essence, looked at 

20 

lack copyrights? 

21 

the topic of the impact of copyright 

21 

MR. FEE: Objection to the 

22 

and trademark infringement here, and 

22 

extent it calls for a legal 

23 

asked myself the question whether a 

23 

conclusion. 

24 

permanent injunction would be 

24 

THE WITNESS: I don't know how 

25 

appropriate from an economic 

25 

to answer that question. I didn't ask 
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1 

Q. Why did you not consider the 

1 

perspective. 

2 

economic effects of free distribution of 

2 

BY MR. BRIDGES: 

3 

standards with respect to other 

3 

Q. And what is the relevance of 

4 

organizations? 

4 

economic analysis to that question, as you 

5 

A. I didn't quite see the 

5 

understand it? 

6 

relevance to this matter. 

6 

MR. FEE: Objection to form. 

7 

Q. Why? 

7 

Vague. Might also be construed to 

8 

A. I don't know how to prove a 

8 

require a legal conclusion. 

9 

negative. 

9 

THE WITNESS: Economists have a 

10 

Q. What's the negative you were 

10 

view and perspective at looking at 

11 

thinking of that would need to be proved or 

11 

issues that some courts have found to 

12 

disproved? 

12 

be useful. 

13 

A. That something is not relevant. 

13 

BY MR. BRIDGES: 

14 

Q. You just didn't see the 

14 

Q. Well, I'm asking, with specific 

15 

relevance? 

15 

relevance to this case, what do you 

16 

A. I don't understand how that 

16 

understand the importance of economic 

17 

would be helpful in the assignment that I had 

17 

analysis to be in this case — 

18 

here. 

18 

MR. FEE: Objection. Calls - 

19 

Q. And what was the assignment you 

19 

BY MR. BRIDGES: 

20 

had here? 

20 

Q. — as you have purported to 

21 

A. Well, I've laid it out — 

21 

practice it? 

22 

Q. I can read the report. I'm not 

22 

MR. FEE: Calls for a legal 

23 

asking you to read — read the report. I'd 

23 

conclusion. 

24 

like your own words now, sitting here. 

24 

Also, to the extent that 

25 

MR. FEE: Objection. 

25 

responding to that would require you 
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1 

BY MR. BRIDGES: 

1 

to disclose communications with 

2 

Q. How do you — how do you 

2 

counsel that did not form the basis 

3 

view — 

3 

for any of your opinions or 

4 

A. I'd like to answer it by 

4 

conclusions and did not provide any 

5 

looking at my report. 

5 

assumptions that were the basis for 

6 

Q. No, I'd like for you to give me 

6 

your opinions or conclusions, you 

7 

a straight answer, because if you're just 

7 

should not answer that portion of the 

8 

going to refer to the report, the report will 

8 

question. 

9 

speak for itself, and I don't need you to 

9 

THE WITNESS: I understand 

10 

read it to me. 

10 

that, generally, economists like me 

11 

I'd like for you to tell me 

11 

are quite helpful in determining 

12 

what you understand, sitting here, to have 

12 

questions of harm, particularly harm 

13 

been your assignment in this case. 

13 

as it relates to infringement of IP 

14 

MR. FEE: Objection. 

14 

rights. 

15 

You can answer the question 

15 

BY MR. BRIDGES: 

16 

however you deem appropriate. 

16 

Q. How do you distinguish between 

17 

THE WITNESS: I've aptly laid 

17 

harms that are caused by an infringement by 

18 

it out in my report, so I defer to the 

18 

the defendant versus harms that might be 

19 

words in my report. 

19 

caused by a court decision that plaintiffs 

20 

But I've, in essence, looked at 

20 

lack copyrights? 

21 

the topic of the impact of copyright 

21 

MR. FEE: Objection to the 

22 

and trademark infringement here, and 

22 

extent it calls for a legal 

23 

asked myself the question whether a 

23 

conclusion. 

24 

permanent injunction would be 

24 

THE WITNESS: I don't know how 

25 

appropriate from an economic 

25 

to answer that question. I didn't ask 
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1 

myself the question of ownership or 

1 

under the assumption that the 

2 

impact of ownership. I asked myself 

2 

activities violate the law. 

3 

the question here of impact of 

3 

BY MR. BRIDGES: 

4 

infringement. 

4 

Q. If the activities — do you 

5 

BY MR. BRIDGES: 

5 

believe — do you understand that your 

6 

Q. If it turns out that the Court 

6 

analysis is relevant to a determination of 

7 

rules that the plaintiff — sorry. Strike 

7 

whether the defendant has violated the law? 

8 

that. 

8 

MR. FEE: Objection. Calls for 

9 

If it turns out the Court rules 

9 

a legal conclusion. 

10 

here that the defendant has engaged in fair 

10 

To the extent that your 

11 

use, is it your understanding that none of 

11 

understanding is based upon 

12 

your harms analysis is relevant — 

12 

communications with counsel, you 

13 

MR. FEE: Objection. 

13 

shouldn't disclose them, unless they 

14 

BY MR. BRIDGES: 

14 

formed the basis for your opinions or 

15 

Q. — because of a finding of 

15 

conclusions or provided assumptions 

16 

non-infringement? 

16 

that you relied upon in reaching your 

17 

MR. FEE: Calls for a legal 

17 

conclusions. 

18 

conclusion. 

18 

THE WITNESS: I don't know. 

19 

To the extent answering that 

19 

BY MR. BRIDGES: 

20 

question would require you to disclose 

20 

Q. Do you have any view as to 

21 

communications you had with counsel 

21 

whether the defendant has violated copyright 

22 

that don't form the basis for any of 

22 

law? 

23 

your opinions or conclusions and don't 

23 

MR. FEE: Objection. Calls for 

24 

provide any assumptions that you 

24 

a legal conclusion. 

25 

relied upon, you shouldn't disclose 

25 

THE WITNESS: No, I've not 
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1 

those communications. 

1 

taken on that assignment. 

2 

THE WITNESS: You're asking for 

2 

BY MR. BRIDGES: 

3 

a legal conclusion. I'm not an expert 

3 

Q. Do you have any view as to 

4 

on that. 

4 

whether the defendant's activities constitute 

5 

BY MR. BRIDGES: 

5 

fair use? 

6 

Q. I'm understanding your 

6 

MR. FEE: Objection. Calls for 

7 

understanding - I'm asking for your 

7 

a legal conclusion. 

8 

understanding of the relevance of your 

8 

THE WITNESS: No, I've not 

9 

contributions to this case. 

9 

taken on that assignment. 

10 

MR. FEE: Objection. Asked and 

10 

BY MR. BRIDGES: 

11 

answered. Plus all the prior 

11 

Q. If a court determines that the 

12 

objections and instructions. 

12 

defendant has not infringed upon plaintiffs' 

13 

THE WITNESS: I believe my 

13 

copyrights, do you understand that the 

14 

testimony and report are relevant to 

14 

decision would result in economic harm to the 

15 

the issue of harm and potential harm. 

15 

plaintiffs? 

16 

BY MR. BRIDGES: 

16 

MR. FEE: Objection to the 

17 

Q. From what? 

17 

extent it calls for a legal 

18 

A. From continuing — the 

18 

conclusion. 

19 

continuing activities and possible expanded 

19 

THE WITNESS: I'm not following 

20 

activities of the defendant here. 

20 

your question. Could you ask it a 

21 

Q. From activities or from 

21 

little bit differently, please? 

22 

violations of law? 

22 

BY MR. BRIDGES: 

23 

MR. FEE: Objection. Vague. 

23 

Q. No, I'll restate it if you just 

24 

Calls for a legal conclusion. 

24 

need to rehear it. 

25 

THE WITNESS: I-I'm working 

25 

A. No, I don't need to rehear it. 
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1 

If you could recast it, please. 

1 

that's fine. 

2 

Q. No. Then please answer my 

2 

A. I want to, but I cannot. 

3 

question. 

3 

Q. Well - 

4 

MR. FEE: Objection. 

4 

A. I do not understand the 

5 

BY MR. BRIDGES: 

5 

question. 

6 

Q. I get to ask the questions. 

6 

Q. I'll say it again. 

7 

MR. FEE: He just said he 

7 

Would a decision by the Court 

8 

couldn't answer it. 

8 

that the defendant has not infringed upon the 

9 

THE WITNESS: I don't 

9 

plaintiffs' copyrights result in economic 

10 

understand the question. 

10 

harm to the plaintiffs? 

11 

BY MR. BRIDGES: 

11 

MR. FEE: Objection. Calls for 

12 

Q. What is it you don't 

12 

a legal conclusion. Asked and 

13 

understand? 

13 

answered. 

14 

A. I understand each word but not 

14 

THE WITNESS: I- 

15 

how you put them together. 

15 

MR. FEE: Vague. 

16 

Q. If a court determines that the 

16 

THE WITNESS: I cannot answer 

17 

defendant has not infringed upon the 

17 

it any differently. I'm sorry. 

18 

plaintiffs' copyrights, do you believe that 

18 

Is this a good time for a 

19 

that decision would result in economic harm 

19 

break, or do you want to keep going? 

20 

to the plaintiffs? 

20 

MR. BRIDGES: Sure. We can 

21 

MR. FEE: Objection to the 

21 

take one if you want. 

22 

extent it calls for a legal 

22 

THE VIDEOGRAPHER: Off the 

23 

conclusion. Plus asked and answered. 

23 

record at 1:17. 

24 

THE WITNESS: It sounds like 

24 

* * * 

25 

exactly the same words, so I'm not 

25 

(Recess from 1:17 p.m. to 
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1 

sure how to answer that question. 

1 

2:12 p.m.) 

2 

BY MR. BRIDGES: 

2 

* * * 

3 

Q. Would a decision that the 

3 

THE VIDEOGRAPHER: On the 

4 

defendant has not infringed upon plaintiffs' 

4 

record at 2:12. 

5 

copyrights result in economic harm to the 

5 

BY MR. BRIDGES: 

6 

plaintiffs? 

6 

Q. Good afternoon, Mr. Jarosz. 

7 

MR. FEE: Objection. Calls for 

7 

A. Good afternoon. 

8 

a legal conclusion. 

8 

Q. Could you outline for me, 

9 

THE WITNESS: I'm just not 

9 

please, what steps you took in your 

10 

following. I under — I'm worked — 

10 

engagement in this case? What are the 

11 

I'm working under the assumption that 

11 

different activities you engaged in? 

12 

the activity here represents a 

12 

A. Generally, I had a discussion 

13 

copyright infringement. I'm - and 

13 

with counsel about the matter. Then we 

14 

I'm being asked and answering the 

14 

examined documents that would — were 

15 

question of the impact of that and 

15 

provided to us to give us background. We 

16 

whether there would be harm and what 

16 

then proceeded to gather our own information 

17 

kind of harm and whether that's 

17 

from third-party sources, primarily through 

18 

reparable harm. 

18 

Internet searches. 

19 

So I'm focusing on what has 

19 

We obtained information that 

20 

been done and what may continue to be 

20 

had been produced as part of discovery. We 

21 

done by the defendant. 

21 

had conversations with people at the various 

22 

BY MR. BRIDGES: 

22 

plaintiff organizations. 

23 

Q. That's non-responsive. I'll 

23 

We outlined the report and 

24 

ask you to answer my question. And if you 

24 

summarized some of the information that you 

25 

just don't want to answer the question, 

25 

see in the tabs. We had discussions with 
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1 

If you could recast it, please. 

1 

that's fine. 

2 

Q. No. Then please answer my 

2 

A. I want to, but I cannot. 

3 

question. 

3 

Q. Well - 

4 

MR. FEE: Objection. 

4 

A. I do not understand the 

5 

BY MR. BRIDGES: 

5 

question. 

6 

Q. I get to ask the questions. 

6 

Q. I'll say it again. 

7 

MR. FEE: He just said he 

7 

Would a decision by the Court 

8 

couldn't answer it. 

8 

that the defendant has not infringed upon the 

9 

THE WITNESS: I don't 

9 

plaintiffs' copyrights result in economic 

10 

understand the question. 

10 

harm to the plaintiffs? 

11 

BY MR. BRIDGES: 

11 

MR. FEE: Objection. Calls for 

12 

Q. What is it you don't 

12 

a legal conclusion. Asked and 

13 

understand? 

13 

answered. 

14 

A. I understand each word but not 

14 

THE WITNESS: I- 

15 

how you put them together. 

15 

MR. FEE: Vague. 

16 

Q. If a court determines that the 

16 

THE WITNESS: I cannot answer 

17 

defendant has not infringed upon the 

17 

it any differently. I'm sorry. 

18 

plaintiffs' copyrights, do you believe that 

18 

Is this a good time for a 

19 

that decision would result in economic harm 

19 

break, or do you want to keep going? 

20 

to the plaintiffs? 

20 

MR. BRIDGES: Sure. We can 

21 

MR. FEE: Objection to the 

21 

take one if you want. 

22 

extent it calls for a legal 

22 

THE VIDEOGRAPHER: Off the 

23 

conclusion. Plus asked and answered. 

23 

record at 1:17. 

24 

THE WITNESS: It sounds like 

24 

* * * 

25 

exactly the same words, so I'm not 

25 

(Recess from 1:17 p.m. to 
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1 

sure how to answer that question. 

1 

2:12 p.m.) 

2 

BY MR. BRIDGES: 

2 

* * * 

3 

Q. Would a decision that the 

3 

THE VIDEOGRAPHER: On the 

4 

defendant has not infringed upon plaintiffs' 

4 

record at 2:12. 

5 

copyrights result in economic harm to the 

5 

BY MR. BRIDGES: 

6 

plaintiffs? 

6 

Q. Good afternoon, Mr. Jarosz. 

7 

MR. FEE: Objection. Calls for 

7 

A. Good afternoon. 

8 

a legal conclusion. 

8 

Q. Could you outline for me, 

9 

THE WITNESS: I'm just not 

9 

please, what steps you took in your 

10 

following. I under — I'm worked — 

10 

engagement in this case? What are the 

11 

I'm working under the assumption that 

11 

different activities you engaged in? 

12 

the activity here represents a 

12 

A. Generally, I had a discussion 

13 

copyright infringement. I'm - and 

13 

with counsel about the matter. Then we 

14 

I'm being asked and answering the 

14 

examined documents that would — were 

15 

question of the impact of that and 

15 

provided to us to give us background. We 

16 

whether there would be harm and what 

16 

then proceeded to gather our own information 

17 

kind of harm and whether that's 

17 

from third-party sources, primarily through 

18 

reparable harm. 

18 

Internet searches. 

19 

So I'm focusing on what has 

19 

We obtained information that 

20 

been done and what may continue to be 

20 

had been produced as part of discovery. We 

21 

done by the defendant. 

21 

had conversations with people at the various 

22 

BY MR. BRIDGES: 

22 

plaintiff organizations. 

23 

Q. That's non-responsive. I'll 

23 

We outlined the report and 

24 

ask you to answer my question. And if you 

24 

summarized some of the information that you 

25 

just don't want to answer the question, 

25 

see in the tabs. We had discussions with 
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1 

counsel. And then we finalized the report, 

1 

Q. Have you analyzed any 

2 

submitting it to counsel on June 5th, 2015. 

2 

differences in sales trends between those of 

3 

Q. Do you know how many standards 

3 

plaintiffs' standards that have been 

4 

of each plaintiff are at issue in this case? 

4 

incorporated into law and those of 

5 

A. How many — I'm sorry — 

5 

plaintiffs' standards that have not been 

6 

standards are at issue? 

6 

incorporated into law? 

7 

Q. Yes. 

7 

A. I don't think so. I don't 

8 

A. I have that number written 

8 

think I have those data, and I'm not sure 

9 

down. It's in the hundreds, and I forget, as 

9 

that each plaintiff knows precisely how many 

10 

I sit here right now, precisely the number. 

10 

have been incorporated into law. 

11 

I will look it up. And I was giving you an 

11 

Q. Did you ask for any data 

12 

answer that was a cumulation across the three 

12 

regarding the distinction between standards 

13 

plaintiffs. 

13 

incorporated by reference and standards not 

14 

I am not seeing that number 

14 

incorporated by reference in the law? 

15 

right now. I'll keep looking. 

15 

A. I don't — 

16 

Q. Do you know what — 

16 

MR. FEE: Objection to form. 

17 

A. You may be able to point me 

17 

THE WITNESS: I'm sorry. I 

18 

quicker than I recall where it was. 

18 

don't recall. 

19 

Q. Do you — do you know what 

19 

BY MR. BRIDGES: 

20 

proportion of plaintiffs — of each 

20 

Q. You made observations about 

21 

plaintiffs' standards is at issue in this 

21 

sales trends earlier in your deposition. I 

22 

case? 

22 

think you said that there's been a reduction 

23 

A. Are you asking me the ratio of 

23 

in sales of certain of plaintiffs' standards; 

24 

the standards at issue versus the total 

24 

is that correct? 

25 

standards developed by the organizations? 

25 

A. I'm not quite sure what the 
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1 

Q. Yes. 

1 

earlier testimony was, but I think I was 

2 

A. I think it's less than a 

2 

pointing you to paragraph 133 with regard to 

3 

majority for each organization. I'm fairly 

3 

downloads of — and other measures of 

4 

certain of that with regard to ASTM. I think 

4 

activity, as I had at my disposal. 

5 

that's true with regard to NFPA. I think 

5 

Q. Well, I'm trying to find out 

6 

it's true with regard to ASHRAE. 

6 

what changes you have studied in plaintiffs' 

7 

Q. Do you have any better 

7 

economics that you attribute to defendant's 

8 

information than less than a majority — 

8 

activities. 

9 

A. Well, I - 

9 

A. I'm not quite sure what your 

10 

Q. — for each of them? 

10 

question is. 

11 

A. The precise numbers are in the 

11 

Q. Well, I'm trying to find out 

12 

report. Let's see here. One can figure that 

12 

what information you have studied to 

13 

out. You may remember where I summarized the 

13 

determine what changes in the finances of 

14 

number of standards. I just don't remember. 

14 

each of the plaintiffs have occurred as a 

15 

It's easy to determine because the data are 

15 

consequence of the defendant's activities. 

16 

all here. 

16 

MR. FEE: Objection to form. 

17 

Q. Have you analyzed differences 

17 

THE WITNESS: I'm still not 

18 

in sales trends between standards that are at 

18 

sure that I'm hearing a question. But 

19 

issue in this case and plaintiffs' other 

19 

to the extent that I had information 

20 

standards? 

20 

on changes in activity level, I 

21 

A. No, I don't think I have those 

21 

summarized that in paragraph 133. 

22 

data at my disposal. 

22 

BY MR. BRIDGES: 

23 

Q. Did you ever ask for those 

23 

Q. My question is, what 

24 

data? 

24 

information did you study to determine any 

25 

A. I don't recall. 

25 

changes in finances of each of the 
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1 

plaintiffs? 

1 

of certain of the standards. I've 

2 

MR. FEE: Same objection. 

2 

presented that. 

3 

THE WITNESS: It's reflected in 

3 

I don't have direct evidence of 

4 

paragraph 133 and in the tabs, 

4 

the precise impact historically of 

5 

particularly 3, 4, and 5. But the 

5 

defendant's activities on plaintiffs' 

6 

tabs are not at the granular level 

6 

financials. 

7 

that I think are of interest to you. 

7 

BY MR. BRIDGES: 

8 

BY MR. BRIDGES: 

8 

Q. What evidence of any kind do 

9 

Q. What do you mean by the 

9 

you have of any kind of impact historically 

10 

"granular level" that would be of interest to 

10 

of the defendant's activities on plaintiffs' 

11 

me? 

11 

financials? 

12 

A. I don't think it breaks out 

12 

MR. FEE: Objection to form. 

13 

publications by standard, for instance. 

13 

THE WITNESS: That which is 

14 

Q. Does it break out publications 

14 

reported in paragraph 133, that of 

15 

by whether a standard has been incorporated 

15 

which is contained in deposition 

16 

by reference or not? 

16 

testimony, and that of which I 

17 

A. I don't think so. 

17 

summarized in other parts of the 

18 

Q. Does it break out by whether a 

18 

report. 

19 

standard has been publicly made available by 

19 

BY MR. BRIDGES: 

20 

defendant or not? 

20 

Q. So when you're referring to 

21 

A. I don't think so. Not in 

21 

deposition testimony, you're referring to the 

22 

tabs 3, 4, and 5. 

22 

citations to the footnotes in paragraph 133? 

23 

Q. How do you establish causation 

23 

A. No, I don't think it's just 

24 

between defendant's activities and any of the 

24 

limited to that. I think there's some other 

25 

data that you provide in section — in 

25 

deposition transcripts that talk about the 
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1 

paragraph 133? 

1 

impact or potential impact of defendant's 

2 

MR. FEE: Objection. Calls for 

2 

activities on each one of the plaintiffs. 

3 

a legal conclusion. Form. 

3 

Q. Did you make any independent 

4 

THE WITNESS: One can and 

4 

assessment of causation of any financial 

5 

should look at all evidence available, 

5 

effects on plaintiffs by the defendant's 

6 

including circumstantial evidence. I 

6 

activities? 

7 

don't have direct information about 

7 

MR. FEE: Objection to form. 

8 

the precise impact of defendant's 

8 

Calls for a legal conclusion. 

9 

activities, but I have important 

9 

THE WITNESS: What do you mean 

10 

information that bears on that issue, 

10 

by the term of "independent assessment 

11 

including information that's in 

11 

of causation"? 

12 

deposition transcripts. 

12 

BY MR. BRIDGES: 

13 

BY MR. BRIDGES: 

13 

Q. You, as an expert, not relying 

14 

Q. So my question is, how do 

14 

just on what other people have said or 

15 

you — do you — strike that. 

15 

speculated or thought. 

16 

Are your conclusion — are you 

16 

MR. FEE: Same objections. 

17 

making conclusions in paragraph 133 about the 

17 

Plus compound. 

18 

cause of changes in sales of the plaintiffs' 

18 

THE WITNESS: We experts rely 

19 

products? 

19 

on other information to draw the 

20 

MR. FEE: Objection to form. 

20 

conclusions that we do, and then we 

21 

THE WITNESS: Not definitively. 

21 

bring our training to it. So our 

22 

I have observations about the 

22 

observations shouldn't be in a vacuum. 

23 

magnitude and trend of the downloads 

23 

BY MR. BRIDGES: 

24 

of — through defendant's sites. I 

24 

Q. But they should be objective, 

25 

have some information on the downloads 

25 

correct? 
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1 

plaintiffs? 

1 

of certain of the standards. I've 

2 

MR. FEE: Same objection. 

2 

presented that. 

3 

THE WITNESS: It's reflected in 

3 

I don't have direct evidence of 

4 

paragraph 133 and in the tabs, 

4 

the precise impact historically of 

5 

particularly 3, 4, and 5. But the 

5 

defendant's activities on plaintiffs' 

6 

tabs are not at the granular level 

6 

financials. 

7 

that I think are of interest to you. 

7 

BY MR. BRIDGES: 

8 

BY MR. BRIDGES: 

8 

Q. What evidence of any kind do 

9 

Q. What do you mean by the 

9 

you have of any kind of impact historically 

10 

"granular level" that would be of interest to 

10 

of the defendant's activities on plaintiffs' 

11 

me? 

11 

financials? 

12 

A. I don't think it breaks out 

12 

MR. FEE: Objection to form. 

13 

publications by standard, for instance. 

13 

THE WITNESS: That which is 

14 

Q. Does it break out publications 

14 

reported in paragraph 133, that of 

15 

by whether a standard has been incorporated 

15 

which is contained in deposition 

16 

by reference or not? 

16 

testimony, and that of which I 

17 

A. I don't think so. 

17 

summarized in other parts of the 

18 

Q. Does it break out by whether a 

18 

report. 

19 

standard has been publicly made available by 

19 

BY MR. BRIDGES: 

20 

defendant or not? 

20 

Q. So when you're referring to 

21 

A. I don't think so. Not in 

21 

deposition testimony, you're referring to the 

22 

tabs 3, 4, and 5. 

22 

citations to the footnotes in paragraph 133? 

23 

Q. How do you establish causation 

23 

A. No, I don't think it's just 

24 

between defendant's activities and any of the 

24 

limited to that. I think there's some other 

25 

data that you provide in section — in 

25 

deposition transcripts that talk about the 
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1 

paragraph 133? 

1 

impact or potential impact of defendant's 

2 

MR. FEE: Objection. Calls for 

2 

activities on each one of the plaintiffs. 

3 

a legal conclusion. Form. 

3 

Q. Did you make any independent 

4 

THE WITNESS: One can and 

4 

assessment of causation of any financial 

5 

should look at all evidence available, 

5 

effects on plaintiffs by the defendant's 

6 

including circumstantial evidence. I 

6 

activities? 

7 

don't have direct information about 

7 

MR. FEE: Objection to form. 

8 

the precise impact of defendant's 

8 

Calls for a legal conclusion. 

9 

activities, but I have important 

9 

THE WITNESS: What do you mean 

10 

information that bears on that issue, 

10 

by the term of "independent assessment 

11 

including information that's in 

11 

of causation"? 

12 

deposition transcripts. 

12 

BY MR. BRIDGES: 

13 

BY MR. BRIDGES: 

13 

Q. You, as an expert, not relying 

14 

Q. So my question is, how do 

14 

just on what other people have said or 

15 

you — do you — strike that. 

15 

speculated or thought. 

16 

Are your conclusion — are you 

16 

MR. FEE: Same objections. 

17 

making conclusions in paragraph 133 about the 

17 

Plus compound. 

18 

cause of changes in sales of the plaintiffs' 

18 

THE WITNESS: We experts rely 

19 

products? 

19 

on other information to draw the 

20 

MR. FEE: Objection to form. 

20 

conclusions that we do, and then we 

21 

THE WITNESS: Not definitively. 

21 

bring our training to it. So our 

22 

I have observations about the 

22 

observations shouldn't be in a vacuum. 

23 

magnitude and trend of the downloads 

23 

BY MR. BRIDGES: 

24 

of — through defendant's sites. I 

24 

Q. But they should be objective, 

25 

have some information on the downloads 

25 

correct? 
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1 

A. Yes. 

1 

A. I took all the data — 

2 

Q. And that means perhaps not 

2 

MR. FEE: Objection. Form. 

3 

relying upon the views of the parties to the 

3 

Objection to form. 

4 

lawsuit alone, but doing independent analysis 

4 

THE WITNESS: I took all this 

5 

and research, correct? 

5 

data into account. That's why I 

6 

MR. FEE: Objection to form. 

6 

reported it here. 

7 

THE WITNESS: I think one can 

7 

BY MR. BRIDGES: 

8 

and should evaluate and consider the 

8 

Q. And the data that you 

9 

views of the parties, but not limited 

9 

identified in the footnotes in 

10 

investigation to that. 

10 

paragraph 134 - sorry — 133? 

11 

BY MR. BRIDGES: 

11 

A. Yes, I considered that 

12 

Q. So what independent analysis 

12 

information. 

13 

and research did you do other than reviewing 

13 

Q. Do you know in what year the 

14 

the views and statements of the parties in 

14 

defendant posted the 2008 version of the 

15 

this case? 

15 

National Electrical Code on its Web site? 

16 

MR. FEE: Objection. Vague. 

16 

A. I don't know with absolute 

17 

THE WITNESS: I reviewed and 

17 

certainty. I do know a number of the alleged 

18 

summarized the data, as you see in 

18 

activities occurred in late 2012. I don't 

19 

133, that I had at my disposal. I 

19 

know if it's specific to that code or not. 

20 

reviewed writings about the impacts. 

20 

Q. Does it matter to your analysis 

21 

And I took important 

21 

exactly when the defendant posted the 2008 

22 

information from the fact that the 

22 

National Electrical Code on its Web site or 

23 

plaintiffs have brought this lawsuit. 

23 

to Internet Archive? 

24 

The plaintiffs don't want this 

24 

A. I would - 

25 

activity to continue. That is 

25 

MR. FEE: Objection to form. 
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1 

revealed preference information that's 

1 

THE WITNESS: I would consider 

2 

quite important. 

2 

that information if I had it, but I 

3 

BY MR. BRIDGES: 

3 

don't have any reason to think that it 

4 

Q. Tell me about what you mean by 

4 

would change any of the conclusions 

5 

repealed — sorry. Strike that. 

5 

that I drew. 

6 

Tell me what you mean by 

6 

BY MR. BRIDGES: 

7 

"revealed preference." 

7 

Q. The timing of when the 

8 

A. What people do often provides 

8 

defendant posted certain matters wouldn't 

9 

information on what their preferences are. 

9 

change your conclusions? 

10 

Q. And so the fact that plaintiffs 

10 

A. Not based on what I know right 

11 

brought this lawsuit has revealed to you that 

11 

now. My understanding is that much of the 

12 

they prefer to bring the lawsuit, correct? 

12 

activity occurred in 2012, the later half of 

13 

MR. FEE: Objection. Vague. 

13 

2012, and I still have the whole body of 

14 

THE WITNESS: Given the cost, 

14 

evidence that I have considered. So I'm not 

15 

they prefer to bring the lawsuit 

15 

sure if the precise timing would change, but 

16 

rather than not bring it, yes. 

16 

I certainly would consider that. 

17 

BY MR. BRIDGES: 

17 

Q. Do you know in what year 

18 

Q. What else — strike that. 

18 

Public.Resource.Org posted the 2011 version 

19 

What are the data you're 

19 

of the National Electrical Code? 

20 

referring to in page — strike that. 

20 

A. Same answer to the question 

21 

What are the data you're 

21 

that you had with regard to the 2008 code. 

22 

referring to in paragraph 133 that you took 

22 

Q. Can you look at the data in 

23 

into account in discussing or analyzing 

23 

your — the tables attached to your report 

24 

effects of defendant's activities on 

24 

and see if that helps refresh your memory as 

25 

plaintiffs? 

25 

to when the defendant posted NEC 2008 and 
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1 

NEC- NEC 2011? 

1 

appropriateness of a permanent 

2 

A. I can look, and I will. 

2 

injunction here. 

3 

No, it doesn't answer that 

3 

BY MR. BRIDGES: 

4 

question, I don't think. 

4 

Q. Is the appropriate of — is the 

5 

Q. Can you make a prediction as to 

5 

appropriateness of a permanent injunction an 

6 

when the defendant posted NEC 2008 and 

6 

economic question? 

7 

NEC 2011, based on the data attached to your 

7 

A. I think, in part, economic 

8 

report in Exhibit 1 ? 

8 

considerations can be and often are taken 

9 

MR. FEE: Objection to form. 

9 

into account in answering that question. 

10 

THE WITNESS: No, I don't 

10 

Q. Is it an economic question? 

11 

think, based on just those data. 

11 

MR. FEE: Objection. 

12 

BY MR. BRIDGES: 

12 

BY MR. BRIDGES: 

13 

Q. Can you make — give an 

13 

Q. That was my question. 

14 

estimate as to when the defendant posted 

14 

MR. FEE: Asked and answered. 

15 

NEC 2008 and NEC 2011, based on the data 

15 

THE WITNESS: Again, in part. 

16 

attached to your report as Exhibit 1 ? 

16 

BY MR. BRIDGES: 

17 

MR. FEE: Same objection. 

17 

Q. The propriety of 

18 

THE WITNESS: No, I don't 

18 

a preliminary - of a - strike that. 

19 

think, based on just that information. 

19 

It's your testimony that the 

20 

BY MR. BRIDGES: 

20 

propriety of a permanent injunction is, in 

21 

Q. Well, just looking at your 

21 

part, an economic question? 

22 

report, can you tell when defendant posted 

22 

MR. FEE: Objection. Asked and 

23 

NEC 2008 and NEC 2011? 

23 

answered. Form. Calls for a legal 

24 

A. My answer hasn't changed. I 

24 

conclusion. 

25 

still don't know precisely when those were 

25 

THE WITNESS: Yes. As I 
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1 

posted. 

1 

understand it, one factor to consider 

2 

Q. But that doesn't make a 

2 

is the reparability or irreparability 

3 

difference to your economic analysis of the 

3 

of harm. I believe, at its core, 

4 

effects of defendant's activities on the 

4 

that's an economic question. 

5 

plaintiffs? 

5 

BY MR. BRIDGES: 

6 

A. Well, I would be curious — 

6 

Q. And what economic theories did 

7 

MR. FEE: Objection to form. 

7 

you rely upon to conclude that, as an 

8 

THE WITNESS: — curious about 

8 

economic matter, a preliminary - strike 

9 

that information, but I don't have any 

9 

that. 

10 

reason to think it would change the 

10 

What economic theories did you 

11 

conclusions that I drew, and that is 

11 

rely upon to conclude that, as an economic 

12 

that a permanent injunction is 

12 

matter, a permanent injunction is appropriate 

13 

appropriate here. 

13 

in this case? 

14 

BY MR. BRIDGES: 

14 

MR. FEE: Same objections. 

15 

Q. Is it your job to determine 

15 

THE WITNESS: I don't know what 

16 

whether a permanent injunction is 

16 

candidates you have in mind for 

17 

appropriate? Is that what you were hired to 

17 

economic theories. 

18 

do? 

18 

BY MR. BRIDGES: 

19 

A. No. 

19 

Q. Whichever ones you relied upon. 

20 

MR. FEE: Objection. Calls for 

20 

A. I - 

21 

a legal conclusion. Form. Compound. 

21 

MR. FEE: Same objections. 

22 

THE WITNESS: I think it's 

22 

THE WITNESS: - used all of my 

23 

ultimately the Court's decision to 

23 

training and applied it to the facts 

24 

make, but I've been asked what my 

24 

of this case and drew the conclusions 

25 

economic view is as to the 

25 

that I did. 
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1 

BY MR. BRIDGES: 

1 

THE WITNESS: We learn about 

2 

Q. And are there any particular 

2 

price theory. We learn about consumer 

3 

aspects of training that you have beyond what 

3 

behavior. We talk — we learn about 

4 

a first-year college student would have 

4 

manufacturer and supplier actions. We 

5 

gotten in a first-year economics course that 

5 

learn about game theory. We learn 

6 

you have brought to bear by applying 

6 

about econometrics. We learn more 

7 

particular economic theories to this case? 

7 

broadly about quantitative methods. 

8 

A. I think my training makes me 

8 

We learn about a variety of aspects of 

9 

who I am and has helped me in assignments 

9 

industrial organization. There are 

10 

like this. I have beyond a first-year-in- 

10 

many things that we learn beyond the 

11 

college understanding of basic economics, but 

11 

first year of economics training. 

12 

they're very important concepts that are 

12 

BY MR. BRIDGES: 

13 

taught and learned in first-year economics. 

13 

Q. No, I'm asking what you brought 

14 

Q. Well, I want to know if there 

14 

to bear in your analysis in this case. 

15 

are any economic concepts beyond first-year 

15 

A. All those. 

16 

economics that you have brought to bear in 

16 

Q. Okay. What aspect of price 

17 

rendering your conclusions in this case. 

17 

theory did you bring to bear in this case? 

18 

MR. FEE: Objection to form. 

18 

A. I don't know how to answer that 

19 

Asked and answered. 

19 

question besides I understand basic price 

20 

THE WITNESS: Generally, there 

20 

theory and have researched it much and 

21 

are, yes. 

21 

applied that to the facts here. 

22 

BY MR. BRIDGES: 

22 

Q. What was the specific 

23 

Q. What economic concepts have you 

23 

application of price theory that you brought 

24 

brought to bear in your report and analysis 

24 

to bear in this case? 

25 

in this case? 

25 

A. I can't be any more specific 
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1 

A. I'm sorry, because I don't know 

1 

than that. I don't understand your question. 

2 

what you mean by "economic concepts." We get 

2 

Q. What aspect of training about 

3 

trained in things like quantitative methods 

3 

consumer behavior did you bring to bear in 

4 

and intermediate microeconomics, in price 

4 

this case? 

5 

theory, in econometrics, in consumer 

5 

A. I can't be any more specific 

6 

behavior. All those things are beyond the 

6 

than saying that. 

7 

first year. I don't know if you're calling 

7 

Q. What aspects of your training 

8 

those economic theories. Your — your 

8 

about game theory have you brought to bear in 

9 

questioning confuses me. 

9 

your work on this case? 

10 

Q. Well, you referred to the 

10 

A. I can't be any more specific 

11 

important concepts in response to my question 

11 

than that. 

12 

to you about particular aspects of training 

12 

Q. What aspects of econometrics in 

13 

that you have beyond what a first-year 

13 

your training have you brought to bear on 

14 

college student would have gotten in a 

14 

this case? 

15 

first-year economics course that you brought 

15 

A. I can't be any more specific 

16 

to bear by applying economic theories to this 

16 

than that. 

17 

case, and your answer refers to very 

17 

Q. What inform — what aspects of 

18 

important concepts that are taught and 

18 

training in qualitative methods have you 

19 

learned. 

19 

brought to bear on this case? 

20 

And so I'm asking you, what 

20 

A. I didn't say "qualitative 

21 

very important economic concepts have you 

21 

methods," and so it may have been mis-keyed 

22 

brought to bear in your analysis of this 

22 

in. I said "quantitative methods." 

23 

case? 

23 

Q. All right. What aspects of 

24 

MR. FEE: Objection to form. 

24 

quantitative methods of your training did you 

25 

Lack of foundation. 

25 

bring to bear on this case? 
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1 

A. I can't be any more specific 

1 

just on this information. 

2 

than that. 

2 

Q. What else would you need? 

3 

Q. What aspect of your training 

3 

A. I don't know, because I think 

4 

regarding aspects of industrial organization 

4 

it's probably a very easy factual question to 

5 

have you brought to bear on this case? 

5 

determine when the downloading first 

6 

A. I can't be any more specific 

6 

occurred, so I don't know why one would need 

7 

than that. 

7 

to back into it. 

8 

Q. But you did bring the theory of 

8 

Q. Well, when — would one be able 

9 

reveal — revealed preferences to bear on 

9 

to use sales trends as a way of identifying 

10 

this case, correct? 

10 

likely effects of a posting of each standard 

11 

A. Yes. 

11 

by the defendant? 

12 

Q. What other economic theories do 

12 

MR. FEE: Objection. Vague. 

13 

you recall bringing to bear on this case? 

13 

Compound. 

14 

MR. FEE: Objection. Asked and 

14 

THE WITNESS: Maybe; maybe not. 

15 

answered. 

15 

BY MR. BRIDGES: 

16 

THE WITNESS: Everything that 

16 

Q. Why do you say "maybe; maybe 

17 

I've — 

17 

not"? 

18 

MR. FEE: And vague. 

18 

A. I just wouldn't think to do it 

19 

Go ahead. 

19 

that way, so I don't know what you exactly 

20 

THE WITNESS: - I've learned 

20 

have in mind. 

21 

in my training, both educational 

21 

Q. Do you associate the posting of 

22 

training and career training. 

22 

standards by defendant with changes in sales 

23 

BY MR. BRIDGES: 

23 

volume of the standards that the defendant 

24 

Q. Can you be more specific than 

24 

has posted? 

25 

that? 

25 

MR. FEE: Objection to form. 
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1 

A. No. 

1 

THE WITNESS: I don't know what 

2 

* * * 

2 

you mean by that question. 

3 

(Jarosz Exhibit 4 marked for 

3 

BY MR. BRIDGES: 

4 

identification.) 

4 

Q. You don't understand the 

5 

* * * 

5 

question? 

6 

BY MR. BRIDGES: 

6 

A. I do not. 

7 

Q. Mr. Jarosz, do you recognize 

7 

Q. Can you correlate the posting 

8 

Exhibit 4 as a document that you produced in 

8 

of standards by defendant with any changes in 

9 

response to a subpoena in this case? 

9 

sales volumes of the standards that the 

10 

A. Yes. 

10 

defendant has posted? 

11 

Q. What is this document? 

11 

MR. FEE: Objection to form. 

12 

A. It appears to be a summary over 

12 

THE WITNESS: I don't think 

13 

the years 2009 through 2013 of dollars and 

13 

I've attempted to compute the 

14 

quantity of NFPA standards that were sold in 

14 

correlation coefficient here 

15 

the marketplace. 

15 

associated with postings. 

16 

Q. Based upon the trends that you 

16 

BY MR. BRIDGES: 

17 

see in this exhibit, can you estimate when 

17 

Q. I'm not asking for a specific 

18 

you believe it is most likely that the 

18 

correlation coefficient. I'm just asking, 

19 

defendant first published — strike that. 

19 

generally, can you correlate the posting of 

20 

Based upon the trends that you 

20 

standards by defendant with any changes in 

21 

see in this Exhibit 4, can you estimate when 

21 

sales volumes of the standards that 

22 

you believe it is most likely that the 

22 

defendants has - that the defendant has 

23 

defendant first posted each of the standards 

23 

posted with reference to Exhibit 4? 

24 

identified here? 

24 

A. I don't know - 

25 

A. I don't think so, not based 

25 

MR. FEE: Objection. Form. 
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1 

THE WITNESS: I don't recall 

1 

Q. Have you determined in any way 

2 

attempting to do that. And I wouldn't 

2 

the dates at which defendant posted various 

3 

necessarily think that the historical 

3 

standards to its Web site or to the Internet 

4 

impact would — is the end of the 

4 

Archive? 

5 

story as to the harm here. 

5 

A. I don't recall doing a separate 

6 

BY MR. BRIDGES: 

6 

analysis of that, no. 

7 

Q. Is historical impact part of 

7 

Q. How did you learn about the 

8 

the story as to the harm here? 

8 

dates at which defendant posted various 

9 

A. Yes. 

9 

standards to its Web site or to Internet 

10 

Q. What — what can you say by 

10 

Archive? 

11 

looking at Exhibit 4 about the historical 

11 

A. I had conversations with 

12 

impact of the posting of the defendant — of 

12 

counsel on that topic, and I may have seen 

13 

the plaintiffs' standards by the defendant? 

13 

that information contained in certain 

14 

A. I don't know that I can say 

14 

documents like the Complaint, but I don't 

15 

much, because I believe the postings largely 

15 

recall. 

16 

occurred in late 2012, and I only have one 

16 

Q. Did you rely upon information 

17 

period after that. 

17 

regarding those dates from conversations with 

18 

Q. If it turns out that 

18 

counsel? 

19 

defendant's postings were well before 2012, 

19 

MR. FEE: In arriving at his 

20 

would that affect your analysis of the trends 

20 

opinions, you're asking? 

21 

in sales data of the plaintiffs' 

21 

MR. BRIDGES: Arriving at his 

22 

publications? 

22 

understanding of the facts. 

23 

MR. FEE: Objection to form. 

23 

THE WITNESS: I don't know that 

24 

Compound. Vague. 

24 

I did, because I don't recall 

25 

THE WITNESS: Maybe. I would 

25 

reporting those specific dates 
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1 

consider that information in 

1 

anywhere in my report. 

2 

conjunction with these data if you 

2 

BY MR. BRIDGES: 

3 

wanted me to. 

3 

Q. Do you recall taking specific 

4 

BY MR. BRIDGES: 

4 

dates into account in analyzing the effect of 

5 

Q. How — what — what would 

5 

defendant's actions? 

6 

change? 

6 

MR. FEE: Objection to form. 

7 

A. I don't know. I haven't done 

7 

Vague. 

8 

that analysis. 

8 

THE WITNESS: I don't recall 

9 

Q. Have you verified the dates on 

9 

one way or the other. 

10 

which plaintiffs — strike that. 

10 

BY MR. BRIDGES: 

11 

Have you verified the dates at 

11 

Q. Do you know how — strike that. 

12 

which defendant posted the various standards 

12 

Do you know how much revenue 

13 

to its Web site or to Internet Archive? 

13 

each plaintiff derives from the standards at 

14 

A. I don't - 

14 

issue in this case? 

15 

MR. FEE: Objection. Vague. 

15 

A. I don't think I know that 

16 

THE WITNESS: I don't recall 

16 

precise number. 

17 

verifying it. 

17 

Q. Did you — did you ever know 

18 

And are you asking did I 

18 

that number? 

19 

separately go out and determine what 

19 

A. I don't think so. 

20 

that date is and see if that was the 

20 

Q. Did you ever know how much 

21 

same as what was represented in the 

21 

revenue each plaintiff derives from standards 

22 

Complaint, for instance? 

22 

that have been incorporated into law? 

23 

BY MR. BRIDGES: 

23 

A. As opposed to those that have 

24 

Q. Yes. 

24 

not been incorporated? Is that — 

25 

A. No, I don't recall doing that. 

25 

Q. Well, I'm - I'm asking about 
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1 

those standards that have been incorporated 

1 

something just north of 50 percent for 

2 

in the law. I'm asking if you know how much 

2 

ASHRAE. 

3 

revenue each plaintiffs derives — each 

3 

BY MR. BRIDGES: 

4 

plaintiff derives from those standards. 

4 

Q. What do you mean by "if you add 

5 

A. I don't - 

5 

in memberships"? 

6 

MR. FEE: Objection. Form. 

6 

A. I'm not - I'm not quite sure 

7 

THE WITNESS: - think I know 

7 

what you're asking me to define. 

8 

that number, and I'm not sure the 

8 

Q. I'm asking you to explain the 

9 

plaintiffs know that number. 

9 

phrase that you just used, "if you add in 

10 

BY MR. BRIDGES: 

10 

memberships." What did that mean? 

11 

Q. Do you know the percentage of 

11 

A. I talked about that in my 

12 

revenue that each plaintiff derives from 

12 

report. Membership fees are a fairly good 

13 

standards that have been incorporated into 

13 

recollect - a fairly good reflection of 

14 

law? 

14 

amount that would have been paid for 

15 

MR. FEE: Objection to form. 

15 

publications. In other words, publication 

16 

THE WITNESS: I don't think I 

16 

fees — it - let me start this over again. 

17 

do, and I don't believe the plaintiffs 

17 

It makes about as much sense to 

18 

do. 

18 

become a member of ASHRAE as it is to buy 

19 

BY MR. BRIDGES: 

19 

some of the individual publications. As a 

20 

Q. Are you aware of any difference 

20 

result, many people choose to become members 

21 

in profitability to plaintiffs between those 

21 

rather than just buying the publication, as I 

22 

standards that have been incorporated into 

22 

understand it. 

23 

law and those standards that have not been 

23 

Q. How did you learn that? 

24 

incorporated into law? 

24 

A. Having knowledge of the - of 

25 

MR. FEE: Objection to form. 

25 

the price difference and through discussions 
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1 

THE WITNESS: I don't believe 

1 

with people at ASHRAE. 

2 

so. 

2 

Q. How did you learn about the 

3 

BY MR. BRIDGES: 

3 

price difference? 

4 

Q. Do you know - strike that. 

4 

A. I don't recall how I learned 

5 

Are you aware of any difference 

5 

it, but I report it in my report based on 

6 

in profitability to plaintiffs between those 

6 

certain documents I've seen. Perhaps I 

7 

standards that defendant has posted to the 

7 

learned it from their Web site. 

8 

Internet and those standards that defendant 

8 

Q. Did you do any surveys of 

9 

has not posted to the Internet? 

9 

ASHRAE members to validate that assumption" 

10 

MR. FEE: Objection to form. 

10 

A. I'm sorry. Validate what 

11 

THE WITNESS: I don't believe 

11 

assumption? 

12 

so. And as with the previous 

12 

Q. About purchase of a membership 

13 

question, I don't think the plaintiffs 

13 

instead of buying the publication. 

14 

have that information at their 

14 

A. I'm not sure that there's an 

15 

disposal. 

15 

assumption in there. My understanding is 

16 

BY MR. BRIDGES: 

16 

that ASHRAE people are of the belief that 

17 

Q. For each plaintiff, what do you 

17 

many people buy membership rather than 

18 

understand to be the percentage of gross 

18 

individual publications. 

19 

revenue from the sale of standards? 

19 

Q. And in your work, did you 

20 

MR. FEE: Objection to form. 

20 

assume that? 

21 

THE WITNESS: I - I've 

21 

A. I didn't assume that. I worked 

22 

reported that in my report. My memory 

22 

on that — under that understanding. 

23 

is that it's something on the order of 

23 

Q. Oh, it's an understanding, but 

24 

66 percent for ASTM and for NFPA. And 

24 

not an assumption? 

25 

if you add in memberships, it's 

25 

A. Yes. 
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1 

Q. Did that understanding make a 

1 

are to copyrighted publications, correct? 

2 

difference to your analysis? 

2 

A. With the exception of number 3, 

3 

A. It was a factual underpinning. 

3 

which refers to copyrighted publications and 

4 

Q. An underpinning, but not an 

4 

memberships. 

5 

assumption? 

5 

Q. Okay. So my question wasn't 

6 

A. It was not an explicit 

6 

about copyrighted publications. My question 

7 

assumption. 

7 

is, what percentage do you understand of 

8 

Q. But it was an underpinning, not 

8 

plaintiffs' revenues comes from the sale of 

9 

an assumption, is your testimony? 

9 

standards at issue in this case? 

10 

MR. FEE: Objection. Asked and 

10 

A. Thank you for that reminder of 

11 

answered. 

11 

what the question is. 

12 

THE WITNESS: Yes. I don't 

12 

I don't think I know that 

13 

know what or why you're arguing with 

13 

precise percentage. 

14 

me on this. 

14 

Q. What percentage of plaintiffs' 

15 

BY MR. BRIDGES: 

15 

revenues, to your knowledge, comes from the 

16 

Q. I'm not arguing. 

16 

sale of standards incorporated into law? 

17 

A. I don't understand. 

17 

A. I don't know that number. 

18 

Q. I'm just trying to understand 

18 

Q. What percentage of plaintiffs' 

19 

your testimony. That's all. So I'm asking 

19 

revenues, to your understanding, comes from 

20 

some follow-up questions. 

20 

the sale of all standards? 

21 

You stated earlier some 

21 

A. I'm sorry. I thought you asked 

22 

percentages of revenue from the sale of 

22 

that question. I thought the immediate one 

23 

standards. Did you mean to be identifying 

23 

before that was standards. 

24 

what you thought were the percentages of 

24 

Q. No. It was standards at issue 

25 

revenue from the sale of standards or from 

25 

in this case. Then - 
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1 

the sale of all publications? 

1 

A. The one before that. 

2 

A. Let me - let me double-check 

2 

Q. - standards incorporated into 

3 

that. 

3 

law. And now it's all standards. 

4 

Well, in the case of ASTM, for 

4 

A. Right. Thank you. 

5 

instance, I believe it's copyrighted 

5 

I don't know that number 

6 

publications. 

6 

either. 

7 

Q. What page are you referring to 

7 

Q. What percentage of 

8 

in your report? 

8 

plaintiffs' - strike that. 

9 

A. Right now I'm looking at 

9 

What dollar value do you 

10 

page 36, but I think I talk about it at other 

10 

associate with the investments that each 

11 

areas. 

11 

plaintiff has made in the development of the 

12 

Q. So page 36, you're talking 

12 

standards at issue in this case? 

13 

about which paragraph? 

13 

A. I don't think I attributed a 

14 

A. Well, right now I was - 

14 

dollar amount to that precise activity, 

15 

Q. 83? 

15 

because I don't know that amount. 

16 

A. - I was looking at 83, but I'm 

16 

Q. What percentage of plaintiffs' 

17 

turning back to, for more reliable 

17 

operating expenses do you associate with the 

18 

information, to paragraph 15, for instance, 

18 

plaintiffs' development of the standards at 

19 

which says in 2014, 67.1 percent of the 

19 

issue in this case? 

20 

revenue was generated by the sale of 

20 

A. I don't think I know that 

21 

copyrighted publications. For NFPA, that 

21 

number. 

22 

information is shown in paragraph 18. And 

22 

Q. What percentage of plaintiffs' 

23 

for ASHRAE, that information is shown in 

23 

operating expenses do you associate with the 

24 

paragraph 22. 

24 

plaintiffs' development of standards 

25 

Q. All three of those references 

25 

incorporated into law? 
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1 

A. I don't think I know that 

1 

Q. Have you ever had access to any 

2 

number. 

2 

information that I've asked in the last 

3 

Q. What percentage of plaintiffs' 

3 

several questions? 

4 

operating expenses do you associate with the 

4 

MR. FEE: Objection to form. 

5 

plaintiffs' development of standards 

5 

THE WITNESS: I don't believe 

6 

generally? 

6 

so. 

7 

A. I don't think I know that 

7 

BY MR. BRIDGES: 

8 

number. 

8 

Q. Do you know whether plaintiffs 

9 

Q. Do you have any estimates of 

9 

prepare standards through joint sponsorship 

10 

any of those numbers that you just said you 

10 

with any other organizations? 

11 

don't think you know? 

11 

MR. FEE: Objection. Vague. 

12 

MR. FEE: Objection to form. 

12 

THE WITNESS: I think I may 

13 

THE WITNESS: Not sitting here 

13 

have seen a reference to that. I 

14 

right now. 

14 

don't know the extent to which it 

15 

BY MR. BRIDGES: 

15 

occurs, but I wouldn't be surprised to 

16 

Q. Did you at one point ever 

16 

be reminded that it does occur. 

17 

determine those numbers? 

17 

BY MR. BRIDGES: 

18 

A. Not that I recall. 

18 

Q. Are you aware of any, as you 

19 

Q. Do you know what percentage of 

19 

sit here? 

20 

the staff or employees of each plaintiff has 

20 

A. Not as I sit here right now, 

21 

worked on the development of standards at 

21 

but I think I'm aware that it has occurred. 

22 

issue in this case? 

22 

Q. Do you know whether plaintiffs 

23 

MR. FEE: Objection to form. 

23 

receive grants, revenue, or stipends from 

24 

THE WITNESS: I don't think I 

24 

governments that use, reference, or adopt 

25 

know that number. 

25 

their standards? 
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1 

BY MR. BRIDGES: 

1 

MR. FEE: Objection to form. 

2 

Q. Do you know what percentage — 

2 

THE WITNESS: There are grant 

3 

do you have an estimate? 

3 

monies that go to NFPA. I don't know 

4 

A. No. 

4 

the source of those grants. I don't 

5 

MR. FEE: Objection to form. 

5 

see a line for grant revenues for the 

6 

THE WITNESS: Not as I sit 

6 

other two organizations. 

7 

here, no. 

7 

BY MR. BRIDGES: 

8 

BY MR. BRIDGES: 

8 

Q. Did you ask any of the 

9 

Q. Do you know what percentage of 

9 

plaintiffs about the revenues or expenses 

10 

the staff or employees of each plaintiff has 

10 

they have specifically attributable to the 

11 

worked on the development of standards 

11 

standards that defendant has posted to the 

12 

incorporated into law? 

12 

Internet? 

13 

MR. FEE: Objection to form. 

13 

MR. FEE: Objection to form. 

14 

THE WITNESS: Not as I sit here 

14 

THE WITNESS: We generally 

15 

right now. 

15 

talked about that topic with each 

16 

BY MR. BRIDGES: 

16 

plaintiff, and I don't think the 

17 

Q. Do you have an estimate? 

17 

plaintiffs know that amount. They 

18 

A. Not as I sit here right now. 

18 

undertake activities that are 

19 

Q. Do you know what percentage of 

19 

standards oriented. They don't know 

20 

the staff or employees of each plaintiff has 

20 

which of those standards will be 

21 

worked on the development of standards in 

21 

incorporated by reference. 

22 

general? 

22 

BY MR. BRIDGES: 

23 

A. Not as I sit here right now. 

23 

Q. Did you — 

24 

Q. Do you have an estimate? 

24 

A. Or which have been. I don't 

25 

A. Not as I sit here right now. 

25 

think they systematically track those. 
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1 

documents, but they provided them as 

1 

A. I looked at some parts of it. 

2 

part of the discovery process. 

2 

I don't recall that I looked at all aspects 

3 

BY MR. BRIDGES: 

3 

of the database. 

4 

Q. Did you ask them for any 

4 

Q. Did you verify how many 

5 

documents that they had not provided? 

5 

standards were incorporated by reference 

6 

A. I think we generally described 

6 

according to that database? 

7 

the kinds of information that we find useful 

7 

A. No, I did not. 

8 

or typically find useful in matters like 

8 

Q. What do you mean by, "This 

9 

this. 

9 

database reports nearly 13,000 instances of 

10 

Q. After you received documents 

10 

incorporation by reference"? 

11 

from plaintiffs' counsel, did you ask them 

11 

A. I don't know what you're asking 

12 

for any more? 

12 

me to define. 

13 

A. That — that's possible. I 

13 

Q. I'm not asking you to define 

14 

don't recall that. 

14 

anything. I'm asking you to explain what you 

15 

Q. You don't recall. Did you- 

15 

meant by that clause, "This database 

16 

do you have any understanding as to the 

16 

reports" — 

17 

dollar value of staff time and expenses that 

17 

A. I'm sorry. rmjust-I'm 

18 

the plaintiffs have incurred in promoting 

18 

going to be just rearranging words a little 

19 

incorporation of their standards into law? 

19 

bit. There were 13,000 times that there was 

20 

MR. FEE: Objection to form. 

20 

incorporation by reference of a standard. 

21 

Lack of foundation. 

21 

I - I don't - I'm sorry. I 

22 

THE WITNESS: I don't think I 

22 

don't understand what your confusion is. 

23 

have that number, no. 

23 

Q. I'm not confused. I'm just 

24 

BY MR. BRIDGES: 

24 

asking you questions. Okay? So please don't 

25 

Q. Do you have an estimate? 

25 

understand - please don't assume that I'm 
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1 

MR. FEE: Same objections. 

1 

confused. I'm trying to understand what you 

2 

THE WITNESS: Not as I sit here 

2 

meant by that. 

3 

now, no. 

3 

You mean separate instances? 

4 

BY MR. BRIDGES: 

4 

You mean separate laws? What do you mean? 

5 

Q. Did you discuss that issue with 

5 

A. Yes. Separate instances slash 

6 

anyone representing the plaintiffs? 

6 

separate laws. 

7 

MR. FEE: Same objections. 

7 

Q. What did you count as an 

8 

THE WITNESS: It's possible, 

8 

instance? 

9 

but I don't recall having that 

9 

A. Mention in a particular law of 

10 

discussion. 

10 

a standard. 

11 

BY MR. BRIDGES: 

11 

Q. Did you or anybody working with 

12 

Q. In paragraph 57 of your report, 

12 

you attempt to determine the number of 

13 

you refer to "thousands of private-sector 

13 

standards that those 13,000 instances of 

14 

standards." Was your sole support for the 

14 

incorporation by reference referred to? 

15 

statement in paragraph 57 the Bremer article 

15 

A. Not entirely. But if you read 

16 

you cited in footnote 88? 

16 

on that — in that same section, it talks 

17 

A. No. You see I discuss and 

17 

about the number of ASTM standards, the 

18 

provide support for that in subsequent 

18 

numbers of - the number of NFPA standards, 

19 

paragraphs in that section. 

19 

and the number of ASHRAE standards. 

20 

Q. And that includes in 

20 

Q. Well, please tell me where it 

21 

paragraph 58? 

21 

refers to the number of standards. 

22 

A. Yes. 

22 

A. It says, "Including more than 

23 

Q. And did you review the 

23 

2,400 instances involving ASTM standards." 

24 

Standards Incorporated by Reference Database 

24 

So you're right. It doesn't 

25 

that you refer to in paragraph 58? 

25 

have the number of standards. It just has 
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1 

mentions of standard. You're absolutely 

1 

Q. What are the governmental 

2 

right. 

2 

functions with respect to driving that you 

3 

Q. And the same thing is true of 

3 

have in mind? 

4 

the NFPA standards and ASHRAE standards? 

4 

A. I don't have any particular 

5 

A. You're absolutely right, yes. 

5 

ones in mind. 

6 

Q. Do you know how many standards 

6 

Q. In paragraph 59, you say, "At 

7 

that database shows as having been 

7 

least 44 states and territories have adopted 

8 

incorporated by reference? 

8 

ASHRAE 90.1 as part of the commercial 

9 

A. Not sitting here right now. 

9 

building energy code." 

10 

One could perhaps look at what I cited to 

10 

Do you see that? 

11 

answer that question, but I don't know right 

11 

A. Yes, I do. 

12 

now. 

12 

Q. And that also has footnote 95 

13 

Q. Do you know whether anyone 

13 

associated with that as well, correct? 

14 

working for you ever did that work to make 

14 

A. Yes, that's correct. 

15 

that determination? 

15 

Q. How do you explain the fact 

16 

A. I don't recall that being done. 

16 

that that reference in footnote 95 shows that 

17 

Q. Paragraph 59, you say, "At the 

17 

those 44 states, in fact, adopted the 

18 

state level, privately-developed standards 

18 

International Energy Conservation Code that 

19 

are incorporated by reference as part of the 

19 

merely has a reference to an option to use 

20 

exercise of a range of governmental 

20 

ASHRAE 90.1? 

21 

functions." 

21 

MR. FEE: Objection. Lack of 

22 

Do you see that? 

22 

foundation. 

23 

A. Yes. 

23 

THE WITNESS: I don't have any 

24 

Q. What do you mean by 

24 

explanation for that. 

25 

"governmental functions" in that statement? 

25 

BY MR. BRIDGES: 
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1 

A. Things that government agencies 

1 

Q. Did you verify that? 

2 

do. 

2 

A. I did not, no. 

3 

Q. And you give a couple of 

3 

Q. Who did? 

4 

examples, but speaking broadly, what are 

4 

A. I'm sorry. Who verified what? 

5 

governmental functions that involve 

5 

Q. On what — on what did you rely 

6 

incorporation by reference of privately 

6 

to make that statement with that footnote? 

7 

developed standards at the state level? 

7 

A. I may not understand your 

8 

MR. FEE: Objection to form. 

8 

question. I relied on what's identified in 

9 

THE WITNESS: I can only answer 

9 

footnote 95. 

10 

generally. Health and human services, 

10 

Q. But you didn't review foot — 

11 

things that are related to that, 

11 

what's in footnote 95, right? 

12 

safety, driving rules and regulation. 

12 

MR. FEE: Objection. Lack of 

13 

Those are among the things that come 

13 

foundation. 

14 

to mind. 

14 

THE WITNESS: I did. 

15 

BY MR. BRIDGES: 

15 

BY MR. BRIDGES: 

16 

Q. What are the governmental 

16 

Q. You — you reviewed that Web 

17 

functions related to health and human 

17 

site? 

18 

services that you have in mind? 

18 

A. Yes. 

19 

A. I don't have any particular 

19 

Q. Personally? 

20 

ones in mind. 

20 

A. Yes, I believe so. 

21 

Q. What are the governmental 

21 

Q. Do you have an explanation as 

22 

functions relating to safety that you have in 

22 

to why the resource cited in footnote 95 

23 

mind? 

23 

actually shows that the 44 states adopted the 

24 

A. I don't have any particular 

24 

International Energy Conservation Code? 

25 

ones in mind. 

25 

MR. FEE: Objection. Lack of 
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1 

Q. What other benefits do 

1 

a particular period. 

2 

plaintiffs gain from incorporation by 

2 

Q. And then you do the same for 

3 

reference of their standards? 

3 

NFPA documents, correct? 

4 

A. I think that generally covers 

4 

A. Yes. 

5 

it. I may be forgetting things that are laid 

5 

Q. What do you calculate as the 

6 

out in my report, but that's what covers it, 

6 

dollar value of harm to the — to ASTM from 

7 

to the best of my memory right now. 

7 

the accesses and downloads that you refer to 

8 

Are we at a good point for a 

8 

in paragraph 133? 

9 

break? 

9 

A. I haven't calculated that harm. 

10 

Q. If you want. Sure. 

10 

Q. Why not? 

11 

A. Thanks. 

11 

A. I'm not sure if I can at this 

12 

THE VIDEOGRAPHER: Off the 

12 

stage. One estimate would be those number of 

13 

record at 3:12. This is the end of 

13 

downloads times the — well, actually, no, 

14 

media unit number 2. 

14 

let me take that back. I just don't know how 

15 

* * * 

15 

to do it. 

16 

(Recess from 3:12 p.m. to 

16 

Q. Can you be certain that these 

17 

3:41 p.m.) 

17 

accesses or down — and downloads referred to 

18 

* * * 

18 

in paragraph 133, in fact, resulted in 

19 

THE VIDEOGRAPHER: On the 

19 

economic loss to ASTM? 

20 

record at 3:41. This is the beginning 

20 

MR. FEE: Objection to form. 

21 

of media unit number 3 in the 

21 

THE WITNESS: Not with absolute 

22 

deposition of John Jarosz. 

22 

certainty, but with reasonable 

23 

* * * 

23 

certainty I can say some — in some 

24 

(Jarosz Exhibit 5 marked for 

24 

number of these instances, it's likely 

25 

identification.) 

25 

the case that the - that the 
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1 

* * * 

1 

information would have been obtained 

2 

BY MR. BRIDGES: 

2 

from ASHRAE in - or ASTM, rather, 

3 

Q. Mr. Jarosz, I've handed you 

3 

in — through legal means. 

4 

Exhibit 5. This is an article that you cited 

4 

BY MR. BRIDGES: 

5 

in your report, correct? 

5 

Q. Would that - in those 

6 

A. Yes, I believe so. 

6 

instances where you say that the information 

7 

Q. Do you recall how this article 

7 

would have been obtained from ASTM through 

8 

came to your attention? 

8 

legal means, can you put a dollar value on - 

9 

A. I do not. 

9 

or even an estimate of the increased revenue 

10 

Q. Is this an article that you 

10 

that ASTM would have gotten from those 

11 

understand to have been published by 

11 

instances where people obtained the 

12 

plaintiff ASHRAE in its journal? 

12 

information from ASHRAE - sorry - from 

13 

A. Yes, that's my understanding. 

13 

AST - 

14 

Q. And this is an article you 

14 

MR. FEE: Object- 

15 

relied upon with respect to the development 

15 

BY MR. BRIDGES: 

16 

of standard 90, which became standard 90.1, 

16 

Q. - from ASTM? 

17 

correct? 

17 

MR. FEE: Objection to form. 

18 

A. Yes. 

18 

THE WITNESS: No, not based on 

19 

Q. In paragraph 133 of your 

19 

the information I have. I don't think 

20 

report, you talk about a number of 

20 

I have any indication of who was doing 

21 

downloads — strike that — you talk about a 

21 

the downloading and why. 

22 

number of documents accessed through Public 

22 

BY MR. BRIDGES: 

23 

Resource's Web site. Do you see that? 

23 

Q. And do you know what 

24 

A. I talk about the number of ASTM 

24 

alternatives persons who were doing the 

25 

documents that are — that were accessed over 

25 

downloading may have had for obtaining the 
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1 information? 

2 A. Not with certainty, because I 

3 don't know who those persons were, but I 

4 would expect one alternative would be to 

5 obtain it properly, directly from ASTM. 

6 Q. Would that have resulted in 

7 more revenue to ASTM? 

8 A. It may have. If they're 

9 materials that were taken improperly that 

10 would have been paid for, then that would 

11 represent a loss of revenue to ASTM. 

12 Q. Do you know whether any of the 

13 persons who obtained this information from 

14 defendant would have paid for the information 

15 from ASTM? 

16 A. No, not with certainty, because 

17 I don't know the identity of the downloaders 

18 or the reasons for their downloading. 

19 Q. Moreover, those persons might 

20 have accessed the standards from ASTM's 

21 reading room for free and with no revenue to 

22 ASTM, correct? 

23 A. You mean in a but-for world? 

24 Had they not done what they actually did, 

25 alternatively they could have gone to the 

Page 214 

1 free reading room? 

2 Q. Right. 

3 A. That's a possibility, yes. 

4 Q. Do you have an understanding as 

5 to why persons would want to download a file 

6 of a standard instead of viewing it at one of 

7 the plaintiffs' reading rooms? 

8 A. Not with absolute certainty, 

9 but I would imagine downloading would allow 

10 more flexibility in referring to the standard 

11 and using it and sharing that information 

12 with others, whereas reading it in — through 

13 an Internet site is somewhat less flexible, 

14 provides less flexibility for the use of that 

15 information. 

16 Q. What did — what do you 

17 understand to be the difference in 

18 flexibility between possession of a download 

19 and access to a standard through a reading 

20 room? 

21 A. Well, I think that a download 

22 typically has a document that's in hard-copy 

23 form. Copies can made — be made of that and 

24 distributed. Reading things just online 

25 doesn't allow for the wide distribution and 

Page 215 


1 more extended use of that document. 

2 Q. Do you have any evidence about 

3 wide distribution of plaintiffs' standards as 

4 a consequence of defendant's actions? 

5 A. I do not. 

6 Q. Have you reviewed any studies 

7 that would allow you to establish any 

8 connection between the number of accesses or 

9 downloads that Public Resource made possible 

10 and any financial harms to the plaintiffs? 

11 MR. FEE: Objection to form. 

12 THE WITNESS: I don't think 

13 I've seen any study on that, no. 

14 BY MR. BRIDGES: 

15 Q. Have you conducted any studies 

16 that would have allowed you to establish any 

17 connection between the number of accesses or 

18 downloads that Public Resource made possible 

19 and any financial harms to the plaintiffs? 

20 MR. FEE: Objection to form. 

21 THE WITNESS: Not other than 

22 what's contained in my report. 

23 BY MR. BRIDGES: 

24 Q. Please turn to page 45, 

25 paragraph 107, which spills into page 108. 

Page 216 

1 MR. FEE: Page 108? 

2 THE WITNESS: I'm sorry. 

3 Page 108 or paragraph? 

4 BY MR. BRIDGES: 

5 Q. I'm sorry. Paragraph — strike 

6 that. 

7 Let me ask you to turn 

8 paragraph 107 on pages 45 to 46. 

9 A. Okay. I'm there. 

10 Q. I just want to make sure I 

11 understand your language correctly at the 

12 bottom of page 45 and the top of page 46. 

13 Is it your opinion that the 

14 copyright that the plaintiffs assert in their 

15 standards drives sales of other publications 

16 other than the standards themselves? 

17 MR. FEE: Objection. Form. 

18 Vague. 

19 THE WITNESS: I think they're 

20 important for driving sales of 

21 publications that embody those 

22 standards. I don't know that I've 

23 drawn a conclusion that it drives the 

24 sale of other products, but that makes 

25 some sense. 
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1 

BY MR. BRIDGES: 

1 

whether plaintiffs have copyright in — 

2 

Q. Well, doesn't that sentence at 

2 

rights in their value-added publications? 

3 

the bottom of 45 and going on to 46 say that 

3 

MR. FEE: Objection. Vague. 

4 

copyright on plaintiffs' standards drive 

4 

THE WITNESS: I would be 

5 

sales of "handbooks that provide commentary 

5 

curious to know that, but I'm not sure 

6 

on the standards by referring to them"? 

6 

of the significance. I don't think it 

7 

A. You haven't read — 

7 

would change my conclusions, but I 

8 

MR. FEE: Objection. 

8 

would be curious to know that. 

9 

Mischaracterizes the document. 

9 

BY MR. BRIDGES: 

10 

THE WITNESS: You haven't read 

10 

Q. Do you know whether 

11 

the whole sentence. I see that 

11 

incorporation into law drives — strike that. 

12 

sentence to which you refer. 

12 

Do you know whether 

13 

BY MR. BRIDGES: 

13 

incorporation into law of plaintiffs' 

14 

Q. Right. I know I haven't read 

14 

standards drives sales of plaintiffs' 

15 

the whole sentence, but didn't I fairly 

15 

standards? 

16 

capture one part of it, which is the sales 

16 

MR. FEE: Objection to form. 

17 

of — strike that — that copyright on 

17 

Vague. 

18 

plaintiffs' standards drives sales of, among 

18 

THE WITNESS: I don't know with 

19 

other things, "handbooks that provide 

19 

absolute certainty, but it would make 

20 

commentary on standards by referring to 

20 

some sense to me. 

21 

them"? 

21 

BY MR. BRIDGES: 

22 

MR. FEE: Same objection. 

22 

Q. Is it your understanding that 

23 

THE WITNESS: I think you have 

23 

it does? 

24 

generally paraphrased it accurately, 

24 

MR. FEE: Same objection. 

25 

yes. 

25 

THE WITNESS: It would make 
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1 

BY MR. BRIDGES: 

1 

some sense to me, yes. 

2 

Q. And that plaintiffs' copyright 

2 

BY MR. BRIDGES: 

3 

protection — this is the top of — strike 

3 

Q. Are you aware that, in some 

4 

that. 

4 

instances, at least one plaintiff uses the 

5 

And turning to the top of 

5 

legal status of its code to promote the sale 

6 

page 46, plaintiffs' copyright protection on 

6 

of handbooks? 

7 

their standards provides plaintiff with a 

7 

MR. FEE: Objection to form. 

8 

competitive advantage with respect to what 

8 

THE WITNESS: I don't know one 

9 

you call value-added publications, correct? 

9 

way or the other. I don't have reason 

10 

A. You've read part of a sentence, 

10 

to dispute it, but there's not a 

11 

but I do see that sentence, yes. 

11 

particular instance that comes to mind 

12 

Q. And I've fairly paraphrased it 

12 

right now. Maybe you have something 

13 

correctly, correct? 

13 

to refresh my memory. 

14 

MR. FEE: Objection to form. 

14 

BY MR. BRIDGES: 

15 

THE WITNESS: I think, 

15 

Q. Can you provide a dollar value 

16 

generally, yes. 

16 

benefit that plaintiffs receive economically 

17 

BY MR. BRIDGES: 

17 

from the incorporation of their standards by 

18 

Q. Do plaintiffs, to your 

18 

reference? 

19 

understanding, have separate copyrights in 

19 

MR. FEE: Objection. Vague. 

20 

those value-added publications, such as 

20 

Form. 

21 

commentaries and handbooks? 

21 

THE WITNESS: I want to make 

22 

A. I don't know. 

22 

sure that I'm understanding. Could 

23 

Q. You don't know? 

23 

you read that back, please? 

24 

A. Correct. I do not know. 

24 

BY MR. BRIDGES: 

25 

Q. Is it important to you to know 

25 

Q. I'll restate it. 
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1 

Can you provide a — can you 

1 

Q. What else? 

2 

put a dollar value, even an estimate, on the 

2 

A. That's what comes to mind. 

3 

economic benefit that plaintiffs receive from 

3 

Q. Anything else? 

4 

incorporation of their standards into law? 

4 

A. Not this moment, no. I guess, 

5 

MR. FEE: Objection to form. 

5 

potentially, when I think some more about it, 

6 

THE WITNESS: I have not. And 

6 

training and seminars, for instance. 

7 

I'm not sure how one would do that, 

7 

Q. Providers of training and 

8 

subject to thinking more about it. 

8 

seminars? 

9 

BY MR. BRIDGES: 

9 

A. Yes. So that's broader than 

10 

Q. At the top of page 46, you say, 

10 

value-added publications, but there are 

11 

"The Plaintiffs' copyright protection on 

11 

potentially alternative providers of training 

12 

their privately-developed standards provides 

12 

and seminars. 

13 

a competitive advantage with regard to the 

13 

Q. In paragraph 109, you say, "In 

14 

sale of these value-added publications as the 

14 

addition to direct sales of copyrighted 

15 

copyright protection limits the ability of 

15 

materials, the Plaintiffs' materials 

16 

others to sell those publications unless they 

16 

associated with their privately-developed 

17 

are unwilling [sic] to compensate the 

17 

standards provide a competitive advantage 

18 

Plaintiffs for such use." 

18 

with regard to the sale of downstream 

19 

MR. FEE: Objection. 

19 

ancillary/complementary services and 

20 

Mischaracterizes the statement. 

20 

products." 

21 

BY MR. BRIDGES: 

21 

Do you see that? 

22 

Q. Is there something unfair about 

22 

A. Yes. That's what I had in 

23 

my characterization of that statement? 

23 

mind. 

24 

A. I think you read it wrong. You 

24 

Q. And who are the competitors you 

25 

read "willing" to read "unwilling" for some 

25 

have in mind in paragraph 109? 
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1 

reason. 

1 

A. I don't know particular names, 

2 

Q. Oh, I'm sorry. Thank you. 

2 

but - at least I don't recall any sitting 

3 

I'll restate the sentence. 

3 

right now — sitting here right now, but I 

4 

"In particular, the Plaintiffs' 

4 

think there are other providers of these 

5 

copyright protection on their 

5 

downstream services and products. 

6 

privately-developed standards provides a 

6 

Q. And please give me examples of 

7 

competitive advantage with regard to the sale 

7 

what you're calling "downstream services and 

8 

of these value-added publications as the 

8 

products." 

9 

copyright protection limits the ability of 

9 

A. Again, seminars and training, 

10 

others to sell those publications unless they 

10 

for instance. 

11 

are willing to compensate the Plaintiffs for 

11 

Q. Anything else? 

12 

such use." 

12 

A. That's what comes to mind right 

13 

Do you see that statement? 

13 

now. 

14 

A. I do, yes. 

14 

Q. Turning to paragraph 110, you 

15 

Q. And the competitive advantage 

15 

state, "I understand that the ability to 

16 

you've identified there, whom do you 

16 

control these downstream products and 

17 

understand to be the competition? 

17 

services is particularly important to the 

18 

A. Other potential providers of 

18 

Plaintiffs here because the barriers to entry 

19 

these so-called value-added publications. 

19 

in the marketplace for downstream products, 

20 

Q. And what - when you say 

20 

such as training and user manuals, are 

21 

"value-added publications," please give me 

21 

relatively low. For example, according to 

22 

more examples of what types of things fall 

22 

Mr. Comstock of ASHRAE, it is relatively easy 

23 

into that category, as you use the term. 

23 

for unauthorized instructors to read a 

24 

A. Examples would be handbooks 

24 

standard and become (or think that they have 

25 

that provide commentary on the standards. 

25 

become) qualified to provide training or 
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1 

guidance on that standard." 

1 

Q. You're just parroting what 

2 

Do you see that? 

2 

Mr. Comstock said, or did you have an 

3 

A. I do, yes. 

3 

independent view? 

4 

Q. What do you understand — what 

4 

A. No, I heard what he said, and 

5 

did you mean by "unauthorized instructors"? 

5 

it made sense to me. 

6 

A. People that have provided or 

6 

Q. So you put it in your report? 

7 

trying to provide services to the marketplace 

7 

A. Yes. 

8 

that have not been explicitly approved by, 

8 

Q. What independent thought or 

9 

for instance, ASHRAE. 

9 

investigation did you do before you put that 

10 

Q. What do you understand the — 

10 

in your report? 

11 

the nature of — strike that. 

11 

MR. FEE: Objection. Vague. 

12 

You called them "instructors," 

12 

Compound. 

13 

correct? 

13 

THE WITNESS: I can't point to 

14 

A. Yes. 

14 

anything in particular. 

15 

Q. Does that mean that you 

15 

BY MR. BRIDGES: 

16 

envision that these persons are providing 

16 

Q. Would a law-school course on 

17 

some kind of instruction? 

17 

the law and regulation of building 

18 

A. Yes. 

18 

construction provide instruction to law 

19 

Q. What instruction do you 

19 

students? 

20 

understand — what instruction did you have 

20 

MR. FEE: Objection. Vague. 

21 

in mind when you referred to "unauthorized 

21 

Calls for speculation. 

22 

instructors"? 

22 

THE WITNESS: I guess it could. 

23 

A. Generally, how best to 

23 

I have a hard time imagining there 

24 

implement standards or provisions of certain 

24 

would be much demand for such a 

25 

standards. 

25 

course, but I'm in general agreement 


Page 226 


Page 228 

1 

Q. What else? 

1 

that that, in concept, could occur. 

2 

A. Nothing else comes to mind 

2 

BY MR. BRIDGES: 

3 

right now. 

3 

Q. Would it be possible to 

4 

Q. Would your understanding of 

4 

envision that, in the course of such 

5 

"unauthorized instructors" include persons 

5 

teaching, a teacher may wish to analyze some 

6 

who were instructing the public as to what 

6 

of plaintiffs' standards that have been 

7 

the standards require? 

7 

incorporated into law as law and as 

8 

MR. FEE: Objection to form. 

8 

regulation? 

9 

Vague. 

9 

MR. FEE: Objection. Calls for 

10 

THE WITNESS: I didn't have 

10 

speculation. Vague. Form. 

11 

that in mind. I guess that's a 

11 

THE WITNESS: I guess that's 

12 

possibility. 

12 

possible, but I would expect a law 

13 

BY MR. BRIDGES: 

13 

professor would be talking about legal 

14 

Q. And would it be relatively easy 

14 

implications, not the technical 

15 

for unauthorized persons like that to read a 

15 

aspects of a standard. I think they 

16 

standard and think that they have become 

16 

might talk about the implication in a 

17 

qualified to provide training or guidance on 

17 

business that's different from a 

18 

that standard? 

18 

vendor business. 

19 

MR. FEE: Objection. Vague. 

19 

BY MR. BRIDGES: 

20 

BY MR. BRIDGES: 

20 

Q. Well, what about the legal 

21 

Q. Is that your understanding? 

21 

implications of a code for contractors? 

22 

A. According to Mr. Comstock, I 

22 

MR. FEE: Objection. 

23 

believe that's correct. 

23 

BY MR. BRIDGES: 

24 

Q. What do you believe? 

24 

Q. Is that — is that fair ground 

25 

A. I have no reason to doubt him. 

25 

for a law professor to discuss with law 
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1 

Q. You can't point to any 

1 

Q. What probability do you assign 

2 

particular investigation or fact that you're 

2 

to the likelihood that you refer to in the 

3 

relying on in paragraphs 117 to 119? 

3 

first sentence of paragraph 121? 

4 

MR. FEE: Objection to form. 

4 

MR. FEE: Objection to form. 

5 

Asked and answered. 

5 

Lack of foundation. 

6 

THE WITNESS: Everything that's 

6 

THE WITNESS: I don't have a 

7 

embedded in Exhibit 1 is, in part, a 

7 

particular quantitative likelihood 

8 

basis for the observations that I draw 

8 

measure. 

9 

in those paragraphs. 

9 

BY MR. BRIDGES: 

10 

BY MR. BRIDGES: 

10 

Q. Can you give an estimate? 

11 

Q. What probability do you assign 

11 

MR. FEE: Same objection. 

12 

to your prediction in the first sentence of 

12 

THE WITNESS: No. 

13 

paragraph 119? 

13 

BY MR. BRIDGES: 

14 

MR. FEE: Objection. Form. 

14 

Q. Turning to paragraph 126, you 

15 

Lack of foundation. 

15 

refer to an "option available to Plaintiffs 

16 

THE WITNESS: I'm not sure that 

16 

to respond to the loss of protection for 

17 

I've used the term "prediction," but I 

17 

incorporated standards." 

18 

wouldn't assign a particular 

18 

Is it your belief that, if the 

19 

quantitative probability. 

19 

plaintiffs lose this case, they will shut 

20 

BY MR. BRIDGES: 

20 

down their creation of new standards? 

21 

Q. Can you give an estimate? 

21 

A. I think that's a possibility. 

22 

A. No. 

22 

Q. What probability do you assign 

23 

Q. Why not? 

23 

to that? 

24 

A. I don't have a basis for that 

24 

MR. FEE: Objection to form. 

25 

estimate. I have reasoning underlying it, 

25 

Lack of foundation. 
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1 

but I don't have a basis to provide a 

1 

THE WITNESS: I don't have a 

2 

quantitative estimate of my level of 

2 

particular quantitative measure of 

3 

confidence. 

3 

probability for that. 

4 

Q. You refer to "uncertainties" in 

4 

BY MR. BRIDGES: 

5 

the second sentence of paragraph 119, 

5 

Q. What's your best estimate? 

6 

correct? 

6 

MR. FEE: Same objection. 

7 

A. I do, yes. 

7 

THE WITNESS: I don't have a 

8 

Q. What probability do you assign 

8 

quantitative best estimate. 

9 

to the likelihood that you refer to with the 

9 

BY MR. BRIDGES: 

10 

word "likely" in the first sentence of 

10 

Q. Is it more or less than 

11 

paragraph 120? 

11 

50 percent? 

12 

MR. FEE: Objection. Form. 

12 

MR. FEE: Same objections. 

13 

Lack of foundation. 

13 

THE WITNESS: I still don't 

14 

THE WITNESS: I don't have a 

14 

have a quantitative estimate. 

15 

particular quantitative measure of 

15 

BY MR. BRIDGES: 

16 

that. And are you referring to my use 

16 

Q. Is it more or less than 

17 

of the term "likely"? 

17 

80 percent? 

18 

BY MR. BRIDGES: 

18 

MR. FEE: Same objections. 

19 

Q. Yes. 

19 

THE WITNESS: Still don't have 

20 

A. Yes, I don't have a particular 

20 

a quantitative estimate. 

21 

quantification of that. 

21 

BY MR. BRIDGES: 

22 

Q. What particular facts are you 

22 

Q. Is it more or less than 

23 

relying on for that paragraph? 

23 

5 percent? 

24 

A. Everything that you see 

24 

MR. FEE: Same objections. 

25 

reported in Exhibit 1. 

25 

THE WITNESS: Still don't have 
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1 

a quantitative estimate. I think that 

1 

to see what alternatives there are among 

2 

there — with reasonable probability I 

2 

standards development organizations currently 

3 

can draw this conclusion, but I can't 

3 

in existence to carry forward the work of 

4 

be any more precise than that. 

4 

plaintiffs if plaintiffs chose to stop 

5 

BY MR. BRIDGES: 

5 

standards development as a result of the loss 

6 

Q. What do you mean, "with 

6 

of this case? 

7 

reasonable probability"? 

7 

MR. FEE: Same objection. 

8 

A. Based on the information that I 

8 

THE WITNESS: Not that I 

9 

have and the training and logic I bring to 

9 

recall, but I am of the understanding 

10 

it, I think there is a — I say with some 

10 

that each SDO has a different charter, 

11 

confidence what I have said here. 

11 

so I don't know that any SDO has an 

12 

Q. And when you say "likely," do 

12 

identical charter to that of any of 

13 

you mean more than 50 percent likely? 

13 

the three plaintiffs. 

14 

A. Not necessarily, no. 

14 

BY MR. BRIDGES: 

15 

Q. Are you aware of other 

15 

Q. Are you aware that these 

16 

standards development organizations active in 

16 

plaintiffs compete with other SDOs in the 

17 

the same field as the plaintiffs? 

17 

creation of standards in particular fields? 

18 

MR. FEE: Objection. Vague. 

18 

MR. FEE: Objection to form. 

19 

Form. 

19 

Vague. 

20 

THE WITNESS: Perhaps you could 

20 

THE WITNESS: What do you mean 

21 

tell me what you have in mind with 

21 

by the term "compete with" in this 

22 

your use of the term "fields." 

22 

context? 

23 

BY MR. BRIDGES: 

23 

BY MR. BRIDGES: 

24 

Q. Well, are you familiar with 

24 

Q. That they consider others 

25 

AHRI? 

25 

rivals for the same market, in part. 
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1 

A. I have perhaps seen reference 

1 

MR. FEE: Objection to form. 

2 

to that. 

2 

Vague. 

3 

Q. Do you know with which of these 

3 

THE WITNESS: I don't recall 

4 

plaintiffs it — do you — do you know what 

4 

seeing reference to that, but my 

5 

field it's in? 

5 

memory is not perfect. 

6 

A. I don't recall, sitting here 

6 

BY MR. BRIDGES: 

7 

right now, no. 

7 

Q. The — in paragraph 131, you 

8 

Q. Are you familiar with NFRC? 

8 

say, "Simply put, freely-distributed, 

9 

A. I may have seen reference to 

9 

unrestricted versions of Plaintiffs' 

10 

that acronym. 

10 

standards that are or could be incorporated 

11 

Q. Do you know what field it's in? 

11 

by reference can be expected to adversely 

12 

A. Not sitting here right now. 

12 

impact the market for Plaintiffs' standards 

13 

Q. Are you familiar with ICC? 

13 

that are incorporated by reference and to 

14 

A. I have seen reference to that. 

14 

displace sales of these standards by the 

15 

I don't recall what it is, sitting here now. 

15 

Plaintiffs - which can be expected to have a 

16 

Q. Do you know whether other 

16 

material adverse effect on Plaintiffs' 

17 

standards developments organizations would be 

17 

revenues." 

18 

in a position to step forward and to continue 

18 

Do you see that? 

19 

the maintenance and preservation and further 

19 

A. Yes. 

20 

development of the standards of plaintiffs 

20 

Q. By "expected," do you mean more 

21 

here if plaintiffs lose this case? 

21 

than 50 percent likely? 

22 

MR. FEE: Objection to form. 

22 

A. Not necessarily. I don't have 

23 

THE WITNESS: I don't know. 

23 

a quantitative assessment of what I mean by 

24 

BY MR. BRIDGES: 

24 

"expected." 

25 

Q. Have you done any investigation 

25 

Q. Do you mean more than 5 percent 
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1 

new in terms of a theory. 

1 

rest of that paragraph? 

2 

Q. Do you have the same answer 

2 

MR. FEE: Objection. Vague. 

3 

with respect to — strike that. 

3 

THE WITNESS: I looked at the 

4 

What facts do you have — 

4 

financial information, and I talked to 

5 

strike that. 

5 

people at the various plaintiffs. 

6 

What facts are you aware of to 

6 

BY MR. BRIDGES: 

7 

disprove — to disprove Mr. Malamud's theory 

7 

Q. You talked to people at the 

8 

that you refer to in paragraph 144? 

8 

various plaintiffs? 

9 

A. Again, it's the same theory 

9 

A. Yes. 

10 

that's being referenced, but there's 

10 

Q. What did you do to verify the 

11 

additional facts; and that is, the downstream 

11 

truth and accuracy of the things that various 

12 

products and services aren't particularly 

12 

plaintiffs said to you in their 

13 

substantial to these plaintiffs and don't 

13 

conversations? 

14 

appear to be enhanced by a lack of copyright 

14 

MR. FEE: Objection to form. 

15 

protection; that is, the plaintiffs have had 

15 

THE WITNESS: I looked at the 

16 

copyright protection and have said — had 

16 

financial information, and I kept my 

17 

some downstream products and services. It's 

17 

eyes and mind open to the information 

18 

hard to imagine that elimination of that 

18 

in the rest of the record to determine 

19 

copyright protection will enhance that 

19 

if it conflicted with what I learned 

20 

business. 

20 

from the company personnel. 

21 

Q. It's hard to imagine, but are 

21 

BY MR. BRIDGES: 

22 

you aware of any studies to disprove 

22 

Q. Whose financial information did 

23 

Mr. Malamud's theory? 

23 

you look at? 

24 

A. No. 

24 

A. All three of the plaintiffs. 

25 

MR. FEE: Objection. Vague. 

25 

It's summarized in tabs 3, 4, and 5. 
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1 

THE WITNESS: I'm sorry. 

1 

Q. Did you look at the financial 

2 

BY MR. BRIDGES: 

2 

information of any entities other than the 

3 

Q. Have you conducted any studies 

3 

plaintiffs? 

4 

to disprove Mr. Malamud's theory? 

4 

A. I looked at Public Resource 

5 

MR. FEE: Same objection. 

5 

financial information. 

6 

THE WITNESS: Not other than 

6 

Q. Apart from Public Resource and 

7 

what's reflected here in Exhibit 1. 

7 

the plaintiffs, did you look at the financial 

8 

BY MR. BRIDGES: 

8 

information of any other entities in making 

9 

Q. What academic literature have 

9 

the assertions that you made in 

10 

you relied upon to criticize Mr. Malamud's 

10 

paragraph 145? 

11 

theory in paragraph 144? 

11 

A. Not in undertaking my 

12 

A. Nothing specific comes to mind. 

12 

assignment here. 

13 

Q. In paragraph 145, you state 

13 

Q. Did you consider the business 

14 

that, "Mr. Malamud's suggestion that the sale 

14 

models of any entities other than the 

15 

of downstream products and services 

15 

plaintiffs and the defendant in making the 

16 

represents an untapped and undeveloped 

16 

statements criticizing Mr. Malamud's theory 

17 

opportunity for the Plaintiffs is incorrect." 

17 

in paragraph 145? 

18 

Do you see that? 

18 

A. Nothing in particular comes to 

19 

A. Yes, I do. 

19 

mind. I understand that there are 

20 

Q. And then you go on and make 

20 

front-loaded business models, but - at DIN, 

21 

some statements for the rest of the 

21 

for instance, but I don't recall undertaking 

22 

paragraph, correct? 

22 

an investigation of the downstream activities 

23 

A. Yes. 

23 

that they have. 

24 

Q. What studies did you engage in 

24 

Q. Did you undertake any 

25 

to determine the facts that you stated in the 

25 

investigation of downstream activities of 
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1 

other US-based standards development 

1 

unable to quantify that with great 

2 

organizations that make their standards 

2 

accuracy. 

3 

freely available to the public? 

3 

BY MR. BRIDGES: 

4 

A. Not that I recall. 

4 

Q. Have you considered any 

5 

Q. Would that have been relevant 

5 

comparable circumstances apart from this case 

6 

to your analysis? 

6 

that would provide guidance for your 

7 

A. It wasn't necessary to do my 

7 

prediction in the last sentence of 

8 

analysis, but I would be curious if I had 

8 

paragraph 146? 

9 

that information. If I — if I had the 

9 

MR. FEE: Objection to form. 

10 

ability to examine that information, I would 

10 

Vague. 

11 

be curious as to what that shows. 

11 

THE WITNESS: I kept my mind 

12 

Q. In paragraph 146, you state, 

12 

and eyes open to that, but I didn't 

13 

"The loss of publications here will likely 

13 

see information of a good comparator. 

14 

reduce the Plaintiffs' sales of those 

14 

BY MR. BRIDGES: 

15 

downstream products and services." 

15 

Q. Did you research whether there 

16 

Do you see that? 

16 

might be good comparators? 

17 

MR. FEE: That's in 146? 

17 

A. I- 

18 

THE WITNESS: Is that the last 

18 

MR. FEE: Same objection. 

19 

sentence you were reading from? 

19 

THE WITNESS: I did in the 

20 

BY MR. BRIDGES: 

20 

sense of reading through the 

21 

Q. Yes. 

21 

literature and information to see if I 

22 

A. Yeah. 

22 

could learn of something that would be 

23 

Q. Paragraph 146. 

23 

a good comparator, but I didn't learn 

24 

A. Yes, I do see that. 

24 

of such comparator. 

25 

Q. Did you mean the loss of 

25 

BY MR. BRIDGES: 
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1 

copyright in the publications here? 

1 

Q. You looked only at the 

2 

A. Certainly the loss of 

2 

information shown in tab 2 to Exhibit 1 ? 

3 

publications, but I believe it would probably 

3 

A. Yes, I think that's right. 

4 

be better to put the loss of copyright in the 

4 

Q. What economic effect are you 

5 

publications as more reflective of the 

5 

aware of to the Blu-ray Disc Association from 

6 

assignment that I undertook here. 

6 

its providing unrestricted access to its 

7 

Q. What probability do you assign 

7 

standard publications for free? 

8 

to the likelihood that you refer to in that 

8 

A. I don't know. I thought you 

9 

sentence? 

9 

had asked that earlier. If not, I apologize. 

10 

MR. FEE: Objection to form. 

10 

Nonetheless, I don't recall knowing the 

11 

Lack of foundation. 

11 

answer to that question or undertaking that 

12 

THE WITNESS: I haven't 

12 

evaluation. 

13 

assigned a quantitative probability to 

13 

Q. Did Blu-ray Disc Association go 

14 

that. 

14 

out of business? 

15 

BY MR. BRIDGES: 

15 

A. I don't think it's out of 

16 

Q. Have you any estimate? 

16 

business, no. 

17 

MR. FEE: Same objections. 

17 

Q. Has it suffered material harm, 

18 

THE WITNESS: I do not. 

18 

to your knowledge, because of unrestricted 

19 

BY MR. BRIDGES: 

19 

access to its standard publications for free? 

20 

Q. Have you any estimate as to the 

20 

A. I don't know. 

21 

magnitude of the likely reduction of 

21 

Q. Do you believe that, on the 

22 

plaintiffs' sales of downstream products and 

22 

theory of revealed preference, Blu-ray Disc 

23 

services? 

23 

Association has determined that unrestricted 

24 

MR. FEE: Same objections. 

24 

access to its standard publications for free 

25 

THE WITNESS: No, I have been 

25 

is in its interest? 
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1 

A. Yes. It's a different entity 

1 

MR. FEE: Objection. Lack of 

2 

than the SDOs here; but for its purposes, it 

2 

foundation. Vague. 

3 

would appear that it's of the belief that 

3 

THE WITNESS: I'm not - I'm 

4 

that's the optimal path to follow. 

4 

not sure that I understand the concept 

5 

MR. BRIDGES: 1 think - I 

5 

of a standard being out of print, so 

6 

think we may pause things now and 

6 

maybe you could help me with that. 

7 

reserve the remainder of our time. 

7 

BY MR. BRIDGES: 

8 

Just a second. Oh, yes. 

8 

Q. Do you know the term "out of 

9 

BY MR. BRIDGES: 

9 

print"? 

10 

Q. Do you believe that the 

10 

A. Generally, I do, yes. 

11 

plaintiffs are harmed when the defendant 

11 

Q. What do you understand it to 

12 

posts a standard that has been incorporated 

12 

mean? 

13 

by reference — let me strike that. 

13 

A. That it's no longer provided in 

14 

Do you believe that plaintiffs 

14 

print form. 

15 

suffer harm from defendant posting a standard 

15 

Q. All right. So what harm do you 

16 

that is not the latest version of the 

16 

understand plaintiffs would suffer if 

17 

standard? 

17 

defendants posted a standard that is out of 

18 

MR. FEE: Objection. Form. 

18 

print? 

19 

Compound. 

19 

MR. FEE: Objection to form. 

20 

THE WITNESS: Potentially, it 

20 

THE WITNESS: Potentially, it 

21 

could cause confusion in the 

21 

could be the harm similar to outdated 

22 

marketplace as to what's the latest 

22 

standards. 

23 

standard, and there may be some 

23 

BY MR. BRIDGES: 

24 

entities out there that are interested 

24 

Q. In other words, confusion in 

25 

in obtaining an earlier standard that 

25 

the marketplace? 
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1 

would be obtaining it free rather than 

1 

A. Potential confusion in the 

2 

through the legal routes established 

2 

marketplace and potentially providing — yes, 

3 

by the plaintiffs. 

3 

that - that would be one form of it. 

4 

BY MR. BRIDGES: 

4 

Q. What other harms do - would 

5 

Q. Have you done any studies to 

5 

you identify from the defendants posting a 

6 

determine what confusion may be likely in the 

6 

standard that is out of print? 

7 

marketplace in that regard? 

7 

A. Nothing else comes to mind this 

8 

MR. FEE: Objection to form. 

8 

moment, but there could be other things 

9 

THE WITNESS: I have not done a 

9 

that - that I'm not thinking of right now. 

10 

likelihood of confusion study, no. 

10 

Q. What harms do you understand 

11 

BY MR. BRIDGES: 

11 

plaintiffs would suffer if a condition of a 

12 

Q. What research have you done as 

12 

standard being incorporated into law is that 

13 

to whether — strike that. 

13 

plaintiffs could not forbid other entities 

14 

What information do you have 

14 

from making that law available widely and 

15 

about what market there is for earlier 

15 

freely to the public? 

16 

versions of standards when there is a newer 

16 

MR. FEE: Objection to form. 

17 

version in the market? 

17 

Incomplete hypothetical. Compound. 

18 

MR. FEE: Objection to form. 

18 

Calls for speculation. 

19 

THE WITNESS: I don't recall 

19 

THE WITNESS: I don't know. 

20 

undertaking specific research on that 

20 

I've not undertaken that assignment. 

21 

topic. 

21 

I've not given that particular 

22 

BY MR. BRIDGES: 

22 

question any thought. 

23 

Q. What harm do you understand 

23 

It seems economically to be 

24 

plaintiffs would suffer if defendants post a 

24 

quite similar to the actions that have 

25 

standard that is out of print? 

25 

occurred here, but I don't know. I've 
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1 not thought about that particular 

2 topic. 

3 MR. BRIDGES: Okay. I thiuk 

4 we'll pause here and reserve the rest 

5 of the time for a later visit with 

6 you, Mr. Jarosz. 

7 Kevin, this is in reliance on 

8 an exchange of correspondence between 

9 Matt and you, I believe. If, for some 

10 reason - well, no. I think that's 

11 all. 

12 Anything else? 

13 MR. FEE: Well, I don’t have 

14 any questions. 

15 Do you guys have any questions? 

16 MR. REHN: Not at this time. 

17 MR. CUNNINGHAM: No. 

18 MR. BRIDGES: Great. Thank 

19 you. 

20 THE WITNESS: Thank you. 

21 THE VIDEOGRAPHER: All right. 

22 Off the record at 4:31. This ends 

23 media unit number 3 and ends testimony 

24 for August 27th, 2015. 

25 * * * 


1 (Witness excused.) 

2 * * * 

3 (Off the record at 4:31 p.m.) 

4 * * * 

5 
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Be confident your electrical work complies with California law. 

1 message 


NFPA - Your NEC Source <nfpa@e.nfpa.org> Tue, Jun 16, 2015 at 12:46 PM 

Reply-To: 


California has adopted the 2011 NECP. Order the 2011 NEC Handbook today and receive FREE Tabs! 


To ensure proper delivery, add nipa@e.nfpa.org to your e-mail address book 
View as webpage 
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ORDER 

TODAY! 
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California has adopted the 
2011 NEC. 

Order the 2011 NEC 
Handbook today and receive a 
FREE set of 2011 NEC Self- 
Adhesive Index Tabs. 

Act quickly — supplies are 
limited! (Set of 96 for a $17.50 
Value) 



LIMITED TIME OFFER! 

FREE 2011 NEC 
Self-Adhesive Tabs 



Increase efficiency and productivity while you apply the 2011 NEC with 
confidence! Order the 2011 NFPA 70® National Electrical Code Handbook today 
and boost your referencing power with FREE NEC Tabs for quick access to vital 
information. 


The 2011 NEC Handbook helps you meet Code, answer questions, and avoid 
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based on real-world scenarios, and gives you the background behind Code 
revisions, so you can work with authority. Expert commentary clarifies concepts 
and the entire 2011 NEC text is included for reference. The Handbook is also 
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UNITED STATES DISTRICT COURT 
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language that will be used in an ASHRAE standard to ASHRAE. SE1MF 143-44 (citing 
Reiniche Decl. 13-14, Exs. 1-2 (each ASHRAE form features a provision stating that the 
member “understand^] that I acquire no rights in publication of such documents”; the provision 
is referred to in the document as a “copyright release”)). 

For the four standards for which ASTM moved for summary judgment, ASTM presented 
evidence that the leader of the group that developed the standard and/or a member of the 
committee that drafted the standard assigned any and all copyrights in their individual 
contributions to ASTM. SUMF 20-24. With respect to the remaining ASTM standards at 
issue, ASTM has produced evidence that over 25,000 members completed membership renewal 
forms every year since 2007, which is as far back as ASTM maintains membership records, 
mostly using the online membership form. Suppl. SUMF ^ 14-16. Although ASTM did not 
request copyright assignments from its members until approximately 2005, the language in the 
assignments it obtained since then retroactively assigned any copyrights that individual 
possessed in any ASTM standard to ASTM. See SUMF f 18. Although Defendant searched 
high and low to identify isolated examples of individuals who may have renewed their 
memberships using other channels or whose membership forms did not include the assignment 
language, Defendant has not identified even one individual who contributed any language that 
appears in any ASTM standard at issue in this case who did not submit a membership form with 
the assignment language. And, as noted, even a single assignment suffices to give ASTM 
sufficient copyright interest to sue. 

b. The Assignments Are Sufficient to Transfer Copyright 
Ownership to Plaintiffs. 

Defendant’s nitpicking of the language of some of Plaintiffs’ assignments is for naught. 

A valid assignment need not “contain an elaborate explanation” or “‘magic words,’ but must 
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UNITED STATES DISTRICT COURT 
FOR THE DISTRICT OF COLUMBIA 


AMERICAN SOCIETY FOR TESTING AND Case No. 1:13-CV-01215-TSC-DAR 

MATERIALS d/b/a/ASTM INTERNATIONAL; 

NATIONAL FIRE PROTECTION 
ASSOCIATION, INC.; and 

AMERICAN SOCIETY OF HEATING 
REFRIGERATING AND AIR-CONDITIONING 
ENGINEERS, INC. 


Plaintiffs, 


v. 

PUBLIC.RESOURCE.ORG, INC., 


Defendant. 


PUBLIC.RESOURCE.ORG, INC., 


Counterclaimant, 


AMERICAN SOCIETY FOR TESTING AND 
MATERIALS d/b/a/ASTM INTERNATIONAL; 

NATIONAL FIRE PROTECTION 
ASSOCIATION, INC.; and 

AMERICAN SOCIETY OF HEATING 
REFRIGERATING, AND AIR-CONDITIONING 
ENGINEERS, INC. 


Counterdefendants. 
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DECLARATION OF STEVE COMSTOCK 

I, Steve Comstock, declare as follows: 

1. Iam currently employed by the American Society of Heating, Refrigerating, and 
Air Conditioning Engineers (“ASHRAE”) as its Director of Publications and Education. I have 
been employed by ASHRAE since 1974. Based on the information known to me as a result of the 
duties and responsibilities of my position, I have personal knowledge of the facts set forth herein 
and could and would testify competently thereto if called as a witness. 

2. As part of my job responsibilities, questions regarding access to ASHRAE 
standards are ultimately directed to me, including questions regarding access to ASHRAE 
standards by individuals with disabilities. 

3. ASHRAE is a non-profit organization that operates with the mission of advancing 
the arts and sciences of heating, ventilating, air conditioning and refrigerating to serve humanity 
and promote a sustainable world. With that in mind, I have made every effort to make 
accommodations for anyone with a disability who wishes to access ASHRAE standards. These 
situations have not arisen often. 

4. In my 31 years serving as the Director of Publications for ASHRAE, I recall only 
two specific examples where individuals requested that ASHRAE make alternate forms of access 
to ASHRAE publications available due to a disability, and in both instances ASHRAE made the 
appropriate accommodation. In 2013, ASHRAE sent a digital copy of an ASHRAE published 
textbook on HYAC systems to a visually impaired student from the Northern Alberta Institute of 
Technology so that the student could employ screen reader software to access the material audibly. 
Similarly, a hearing impaired individual alerted ASHRAE that he wished to attend a training class 
related to HVAC design, and ASHRAE provided sign-language interpretation. 

1 
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5. ASHRAE has also undertaken additional efforts to ensure that disabilities do not 
unnecessarily limit access to our standards or other services that ASHRAE provides. Last year, 
ASHRAE removed encryption from the digital copies of standards sold on the ASHRAE bookstore 
so that the standards would be more compatible with reading software used by visually impaired 
individuals. ASHRAE’s partner in running the ASHRAE bookstore, a company called Techstreet, 
has made assurances to ASHRAE that it would also help accommodate individuals with 
disabilities. And, ASHRAE has formally adopted a policy allowing for alternate testing 
acco mm odations related to certification programs run by ASHRAE; a request form for test takers 
which to receive such accommodations can be found on the ASHRAE website at 
https://www.ashrae.org/education—certification/certification/faqs#3. 

6. ASHRAE has consistently provided accommodation to individuals with disabilities 
in the past and intends to continue to do so in the future. 

7. I am attaching to this declaration as Exhibit 1 a true and correct copy of ASHRAE 
Standard 90.1-2004, which I understand to be one of the ASHRAE standards at issue in this case. 

In my role as Director of Publications, I am familiar with ASHRAE’s standards, including 90.1. I 
have reviewed this document and it is an accurate copy of Standard 90.1-2004. 

I declare under penalty of perjury under the laws of the United States that the foregoing is 
true and correct. 

Executed this 21 st day of January, 2016 at Orlando, Florida. 



Steve Comstock 
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EXHIBIT 1 
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Filed Under Seal 


JA1939 


Case l:13-cv-01215-TSC Document 155-6 Filed 01/21/16 Page lot5 
USCA Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 181 of 573 


UNITED STATES DISTRICT COURT 
FOR THE DISTRICT OF COLUMBIA 


AMERICAN SOCIETY FOR TESTING 
AND MATERIALS d/b/a/ ASTM 
INTERNATIONAL; 

NATIONAL FIRE PROTECTION 
ASSOCIATION, INC.; and 


AMERICAN SOCIETY OF HEATING, 
REFRIGERATING, AND AIR 
CONDITIONING ENGINEERS, 


Case No. l:13-cv-01215-TSC 


Plaintiffs/ 

Counter-Defendants, 


PUBLIC.RESOURCE.ORG, INC., 

Defendant/ 

Counter-Plaintiff. 


DECLARATION OF CHRISTIAN DjJBAY 
IN SUPPORT OF PLAINTIFFS’ MOTION FOR SUMMARY JUDGMENT 


I, Christian Dubay, declare as follows: 

1. lam Vice President, Codes and Standards, and Chief Engineer for the National 
Fire Protection Association (“NFPA”). My duties include managing and administering the 
NFPA Codes and Standards process. I have held this position since 2007. The following facts 
are based upon my own personal knowledge, and if called upon to do so, I could and would 
testify competently hereto. 

2. A central component of NFPA’s mission is to eliminate the risk of death, injury, 
property and economic loss due to fire, electrical and related hazards, for all people. As part of 
that mission, NFPA has long been involved with developing strategies and fire safety educational 
materials for people with disabilities. 
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3. Since at least 2007, NFPA has had a Disability Access Review and Advisory 
Committee. This committee is appointed by NFPA’s president and advises NFPA’s president 
and its Technical Committees. 

4. The Disability Access Review and Advisory Committee works to identify existing 
needs and emerging issues within the disability community, and to ensure that the NFPA Codes 
and Standards process includes current subject matter that addresses disability issues, access 
provisions, and other matters that impact the disability community. 

5. NFPA has taken a leading role in promoting building safety for the disabled by, 
among other things, developing an Emergency Evacuation Planning Guide for People with 
Disabilities, which is available for free download on NFPA’s website. This Guide provides 
information on the five general categories of disabilities (mobility, visual, hearing, speech, and 
cognitive) and the four elements of evacuation information that occupants need: notification, 
way finding, use of the way, and assistance. 

6. NFPA is also committed to providing access to its standards to all persons who 
have an interest in reading them. As part of that commitment, NFPA makes accommodations for 
disabled persons who request assistance in accessing any of NFPA’s standards. NFPA is not 
aware of any persons who have requested assistance in accessing NFPA materials and have been 
unable to do so. 

7. Iam aware of one instance in which NFPA received a request for accommodation 
in accessing an NFPA standard from a person who had low vision. NFPA responded by 
providing that individual with a PDF copy of the requested standard, free of charge, and the 
individual was able to use that PDF copy to read the standard. 
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8. Attached hereto as Exhibit A is a true and correct copy of the 2011 edition of 
NFPA 70, the National Electrical Code. 

I declare under penalty of perjury under the laws of the United States that the foregoing is 
true and correct and that this declaration was executed this 21st day of January 2016 at Quincy, 
Massachusetts. 
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EXHIBIT A 
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UNITED STATES DISTRICT COURT 
FOR THE DISTRICT OF COLUMBIA 


AMERICAN SOCIETY FOR TESTING 
AND MATERIALS d/b/a/ ASTM 
INTERNATIONAL; 

NATIONAL FIRE PROTECTION 
ASSOCIATION, INC.; and 


AMERICAN SOCIETY OF HEATING, 
REFRIGERATING, AND AIR 
CONDITIONING ENGINEERS, 


Case No. l:13-cv-01215-TSC 


Plaintiffs/ 

Counter-Defendants, 


v. 

PUBLIC.RESOURCE.ORG, INC., 

Defendant/ 

Counter-Plaintiff. 


SUPPLEMENTAL DECLARATION OF THOMAS B. O’BRIEN. JR. 

Pursuant to 28 U.S.C. § 1746,1, Thomas B. O’Brien, Jr., declare the following statements to be 
true under the penalties of peijury: 

1. Iam over the age of 18 years and am fully competent to testify to the matters 
stated in this Declaration. 

2. This declaration is based on my personal knowledge. If called to do so, I would 
and could testily to the matters stated herein. 

3. I am Vice President and General Counsel at ASTM International (“ASTM”). I 
have worked at ASTM since 2003. 

4. Prior to joining ASTM in 2003,1 worked as outside counsel for ASTM between 
1997 and 2003. 
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5. Attached as Exhibit 1 hereto is a true and correct copy of ASTM’s online new 
membership form, which has been in place since 2005. 

6. As shown in Exhibit 1, since 2005, new members to ASTM who completed their 
membership application online had to affirmatively click on a check box next to the following 
statement: “I agree, by my participation in ASTM and enjoyments of the benefits of my annual 
membership, to have transferred and assigned any and all interest I possess or may possess, 
including copyright, in the development or creation of ASTM standards or ASTM IP to ASTM.” 

7. Attached as Exhibit 2 hereto is a true and correct copy of ASTM’s online 
membership renewal form, which has been in place since 2005. 

8. Attached as Exhibit 3 hereto is a true and correct copy of instructions for 
registering a work item through ASTM’s online system, which provides screen shots of each of 
the different screens a member will see when registering a work item. 

9. ASTM has had a version of its “Form and Style for ASTM Standards” (“ASTM 
Form and Style Guide”) since at least as early as 1957. 

10. Attached as Exhibit 4 hereto is a true and correct copy of the version of the 
ASTM Form and Style Guide titled “Recommendations on Form of ASTM Standards,” which 
was published in 1961 and references issuance in 1957. 

11. Each version of the ASTM Form and Style Guide described certain components 
and provided the text for certain language that was required to be included in every ASTM 
standard during the relevant time period. 

12. As part of the process of developing a draft standard, ASTM staff members added 
language and components that were required by the relevant ASTM Form and Style Guide to the 
draft prepared by the task group. 
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13. I have given training to ASTM employees and committee officers on use of the 
ASTM Form and Style Guide in connection with standards, in conjunction with Regulations 
Governing ASTM Technical Committees. 

14. I have attended ASTM committee meetings in which the requirement to use 
certain language and information from the ASTM Form and Style Guide was discussed. 

15. I supervise the ASTM employees who respond to requests to grant permissions to 
use ASTM’s copyrighted materials, and I have personal knowledge of the circumstances and 
frequency with which these requests are granted and denied. 

16. ASTM denies requests for permission to use its standards at no cost when the 
requester seeks to post the standard on a public website with no reasonable time limit and/or with 
no limitation on the number of people who can access it 

17. lam not aware of any visually-impaired person who has informed ASTM that 
he/she was having difficulty accessing an ASTM standard due to a print disability. If a visually- 
impaired person requested access to an ASTM standard that was necessary due to a print 
disability, I would instruct the staff member who received the request to provide a copy of the 
ASTM standard in a format that accommodated the person’s disability at no additional cost to 
the requester. 

18. ASTM’s practice was to obtain a copyright registration for every annual Book of 
Standards from 1980-2011.1 am not aware of any circumstance in which ASTM deviated from 
this practice. 

19. ASTM maintains records related to each ASTM standard that is proposed. Those 
records include information about the standard number, the committee that has jurisdiction over 
the standard, ballot items related to the standard, and the name of the technical contact for the 
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standard. These records are kept in the ordinary course of ASTM’s regularly conducted activity 
at or near the time at which any activities related to the standard took place by a person with 
knowledge of the activities related to the standard. I am familiar with these computer-stored 
records because I use these records to prove legal advice to ASTM. I recognize the documents 
referenced in paragraphs 20-23 below to be printouts from these computer-stored records and the 
printouts accurately reflect the computer-stored records. 

20. Attached as Exhibit 5 is a true and correct printout from the computer-stored 
records described in paragraph 19 above with information regarding ASTM D86-07. 

21. Attached as Exhibit 6 is a true and correct printout from the computer-stored 
records described in paragraph 19 above with information regarding ASTM D975-07. 

22. Attached as Exhibit 7 is a true and correct printout from the computer-stored 
records described in paragraph 19 above with information regarding ASTM D396-98. 

23. Attached as Exhibit 8 is a true and correct printout from the computer-stored 
records described in paragraph 19 above with information regarding ASTM D1217-98. 

24. ASTM maintains records related to members who complete new membership and 
membership renewal forms each year. Those records include information such as the name of 
the member, the date on which the member completed the membership form, and for some of the 
members, whether the member completed the membership through ASTM’s online system, a 
paper form, or another method. These records are kept in the ordinary course of ASTM’s 
regularly conducted activity at or near the time at which the membership forms were completed 
by a person with knowledge of the completion of the membership forms. I am familiar with 
these computer-stored records because I use these records to prove legal advice to ASTM. I 
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recognize the documents referenced in paragraphs 25-26 below to be printouts from these 
computer-stored records and the printouts accurately reflect the computer-stored records. 

25. Attached as Exhibit 9 is a true and correct printout from the computer-stored 
records described in paragraph 24 above showing ASTM individual membership forms that were 
completed in 2007. 

26. Attached as Exhibit 10 is a true and correct printout from the computer-stored 
records described in paragraph 24 above showing ASTM organizational membership forms that 
were completed in 2007. 

Dated: January 21,2016 
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1 


Q And what type of entities are authorized 


2 entities? 


3 


A I don't have the precise code cite, but 


4 authorized entities have to meet some 

5 qualifications, nonprofit or government agency, 

6 primary mission to serve people with disabilities. 

7 It has to be one of their -- one of their primary 

8 missions. There may be some other qualifications, 

9 but those are the big ones I think of. 

10 Q So if you meet those two qualifications, 

11 what exception are you provided? 

12 MR. KAPLAN: Objection; vague, incomplete 

13 hypothetical, calls for a legal conclusion. 

14 THE WITNESS: So some of the provisions of 

15 Chafee, as I recall, are that you can make an 

16 accessible copy of a literary work with some 

17 exceptions for people with qualifying print 

18 disabilities. 

19 BY MS. RUBEL: 

20 Q And you're permitted to make accessible 

21 copies exclusively for people with disabilities; is 

22 that right? 

23 MR. KAPLAN: Objection; argumentative, 

24 calls for a legal conclusion, incomplete 

25 hypothetical, vague. 
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THE WITNESS: The word "exclusively" 
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2 

probably appears in the statute, but I'm not 


3 

100 percent sure. 


4 

BY MS. RUBEL: 


5 

Q Well, is it your understanding that under 


6 

the Chafee Amendment if you meet certain 


7 

requirements, you're permitted to make copies of the 


8 

literary work and distribute them to anyone? 


9 

MR. KAPLAN: Objection; calls for a legal 


10 

conclusion, argumentative, incomplete hypothetical. 


11 

vague. 


12 

THE WITNESS: As an organization that is 


13 

availing itself of the Chafee Amendment, among other 


14 

things, we go to some length to make sure that 


15 

people with print disabilities are the only people 


16 

that are eligible for our service. 


17 

BY MS. RUBEL: 


18 

Q And do you go to those lengths because 


19 

it's your understanding that the Chafee Amendment 


20 

requires you to only make the materials accessible 


21 

to people with print disabilities? 


22 

MR. KAPLAN: Objection; calls for a legal 


23 

conclusion, vague. 


24 

THE WITNESS: As someone who operates 


25 

under the Chafee Amendment to support that, we need 



TOjfef> EPO 

www.CapitalReportmgCompany.com © 2015 



Case l:13-cv-01215-TSC c ^^i^a 
USC A c mffi ;7§gg ety 




1 

to ensure that we only distribute them to people who 
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2 

have qualifying disabilities. Yes. 


3 

BY MS. RUBEL: 


4 

Q Why is that? 


5 

A Because -- 


6 

MR. KAPLAN: Objection; calls for a legal 


7 

conclusion, calls for speculation, vague. 


8 

THE WITNESS: I — I believe that's the 


9 

language of the statute, that it's — that it's 


10 

making the materials available for people with 


11 

disabilities. 


12 

BY MS. RUBEL: 


13 

Q And I think you mentioned that Benetech 


14 

operates as a nonprofit that you would consider an 


15 

authorized entity under the Chafee Amendment? 


16 

MR. KAPLAN: Objection; misstates 


17 

testimony, calls for a legal conclusion, vague. 


18 

THE WITNESS: Yes. 


19 

BY MS. RUBEL: 


20 

Q Are there any other requirements that 


21 

Benetech must meet in order to provide copies of 


22 

literary works to people with print disabilities 


23 

under the Chafee Amendment? 


24 

MR. KAPLAN: Objection; calls for a legal 


25 

conclusion, vague. 
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1 


Those are the three major ways that people 


2 provide proof of disability. 


3 


Q Would that be some kind of doctor? 


4 


MR. KAPLAN: Objection; vague. 


5 


THE WITNESS: It varies by disability what 


6 professional credential someone needs to have to 

7 provide a certification of disability. 

8 BY MS. RUBEL: 


9 


Q So to access materials from Bookshare that 


10 are made pursuant to the Chafee Amendment, an 

11 individual would have had to show some proof of 

12 disability by one of the methods that you just 

13 described; is that correct? 


MR. KAPLAN: Objection; misstates 


14 


15 testimony, vague. 


16 


THE WITNESS: If the Chafee Amendment is 


17 one of the mechanisms we're using to deliver the 

18 book, then there has to be an association with a 

19 person with a qualifying disability under the Chafee 

20 Amendment to obtain a copyrighted work. Yes. 

21 BY MS. RUBEL: 

22 Q And you have that configured into the 

23 software so that the person clicks to open that 

24 work, you will have ensured that you have proof of 

25 disability on file? 
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1 


MR. KAPLAN: Objection; argumentative 


2 vague, lacks foundation. 


3 


THE WITNESS: A user who's logged in who 


4 chooses to download a copyrighted work where we're 

5 using the Chafee Amendment as one of our 

6 justifications for doing so, there is an electronic 

7 indication in -- that that is something they're 

8 permitted to do. 

9 So someone who has not provided proof of 

10 disability is not allowed to download a copyrighted 

11 work under the Chafee Amendment, but they could 

12 download a public domain or creative comments work. 

13 BY MS. RUBEL: 

14 Q If it was a copyrighted work and they 

15 attempted to download it, they did not have proof of 

16 disability on file, what would happen? 

17 MR. KAPLAN: Objection; incomplete 

18 hypothetical, lacks foundation. 

19 THE WITNESS: My understanding is if they 

20 have not provided proof of disability, the 

21 "Download" button does not appear on the title page, 

22 so they can find out that we have the work, but they 

23 can't download it unless their account has that 

24 enabled, and then there's a button that shows up 

25 that says. How do you want to download it? 
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1 MR. KAPLAN: Objection; calls for 

2 speculation, lacks foundation, vague. 

3 THE WITNESS: My understanding was that 

4 one of the uses of the scanned books was to make 

5 books accessible to people with disabilities. 

6 BY MS. RUBEL: 

7 Q Do you know if that consortium had any 

8 safeguards in place that ensured that only people 

9 with print disabilities would be able to access the 

10 copies of those books? 

11 MR. KAPLAN: Objection; vague, lacks 

12 foundation, calls for speculation. 

13 THE WITNESS: I believe that only faculty, 

14 students and staff of the research universities were 

15 able to access information about the books so that 

16 that access control was the primary access control. 

17 I know less about the details of 

18 disability-specific access, but I do believe that 

19 there was some difference between regular, 

20 nondisabled faculty, staff and students and disabled 

21 faculty, staff and students. 

22 BY MS. RUBEL: 

23 Q Do you believe there was some sort of 

24 certification required to show that the person had a 

25 print disability to get access to certain of the -- 
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1 of the books? 


2 


MR. KAPLAN: Objection; vague. 


3 


THE WITNESS: There are a lot of research 


4 libraries that were involved in the case, and I 

5 don't know what their process was beyond saying 

6 these people have print disabilities so they'll get 

7 more extensive access to the works than regular 

8 faculty and staff and students. 

9 BY MS. RUBEL: 

10 Q Are you familiar with -- 

11 Aside from the consortium and the 

12 HathiTrust and Benetech, are you familiar with other 

13 organizations that provide access to copies of 

14 copyrighted works under the Chafee Amendment? 

15 MR. KAPLAN: Objection; vague. 

16 THE WITNESS: There are some national 

17 organizations that are well-known in the field: The 

18 National Library Service for the Blind, Visually 

19 Impaired and Physically Disabled of the Library of 

20 Congress; NLS is the largest. Learning Ally, 

21 formerly known as Recording for the Blind and 

22 Dyslexic; the American Printing House for the Blind, 

23 National Braille Press. 

24 Those would probably be the four 

25 organizations most often cited, along with -- sorry. 
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1 including Bookshare, which is the library that my 

2 nonprofit, Benetech, operates. 

3 But there are other organizations, many 

4 other organizations, that I believe would assert 

5 that they operate under the Chafee Amendment. 

6 BY MS. RUBEL: 

7 Q NLS, do they have some sort of requirement 

8 that an individual provide proof of disability 

9 before being able to access copyrighted materials 

10 under the Chafee Amendment? 

11 A Yes. 

12 MR. KAPLAN: Objection; lacks foundation. 

13 BY MS. RUBEL: 

14 Q What are their requirements? 

15 A They are stated on the NLS website, and 

16 they're similar to those that we use, and we have a 

17 agreement with NLS that if someone has submitted NLS 

18 their qualifications, we accept that as proof of 

19 disability for Bookshare services. 

20 Q What about Learning Ally; do they have a 

21 requirement that the person certify that they have a 

22 print disability before being able to access the 

23 materials? 

24 MR. KAPLAN: Objection; vague, lacks 

25 foundation. 
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1 


THE WITNESS: Yes. 


2 BY MS. RUBEL: 


3 


Q How about the American Printing House for 


4 the Blind; do they have the same requirement? 


5 


MR. KAPLAN: Objection; vague, lacks 


6 foundation. 


7 


THE WITNESS: Their requirement is 


8 different because they're more narrowly focused on 

9 blind and visually impaired students, and I'm not 

10 sure that if they're providing a Braille copy of a 

11 book, that they require people to prove that they're 

12 disabled, because Braille is not -- hard copy 

13 Braille is not easy to make copies of. 

14 BY MS. RUBEL: 

15 Q So they're not providing something, for 

16 example, that could be read by a screen reader? 

17 MR. KAPLAN: Objection; misstates the 

18 testimony, argumentative, lacks foundation, vague. 

19 THE WITNESS: They do. They have some 

20 kind of registration system for students who have 

21 the visual impairments that their organization 

22 serves, and I believe that the visually impaired 

23 students they serve would generally be understood as 

24 qualifying under Chafee. 

25 
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1 

BY MS. RUBEL: 

88 

2 

Q So they may not have a certification 


3 

requirement for people who are blind; is that your 


4 

understanding? 


5 

MR. KAPLAN: Objection; argumentative, 


6 

misstates testimony, vague. 


7 

THE WITNESS: I don't understand all of 


8 

their process, but I would say that their 


9 

identification of visually impaired students through 


10 

school systems and state education agencies is 


11 

comparable to our seeking a proof of disability from 


12 

the school systems, because the school systems are 


13 

legally obligated to serve blind and visually 


14 

impaired students. 


15 

But I did provide proviso that I could 


16 

imagine they might serve up Braille without a proof 


17 

of disability. It's not uncommon in our field that 


18 

hard copy Braille is circulated more broadly, and no 


19 

publisher has ever objected to that. 


20 

BY MS. RUBEL: 


21 

Q Okay. I understand. 


22 

And what about the National Braille Press; 


23 

are they providing things only in Braille? 


24 

MR. KAPLAN: Objection; lacks foundation. 


25 

vague. 
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1 

THE WITNESS: It's my understanding they 

89 

2 

primarily provide Braille books, but they may 


3 

provide other things. I think of them as The 


4 

Braille Press, so... 


5 

BY MS. RUBEL: 


6 

Q Do you know if they have a certification 


7 

requirement to show that you have a print 


8 

disability? 


9 

MR. KAPLAN: Objection; vague. 


10 

THE WITNESS: I don't — sorry. 


11 

(Reporter clarification) 


12 

THE WITNESS: They might. 


13 

BY MS. RUBEL: 


14 

Q Prior to this case -- 


15 

Prior to being retained as an expert in 


16 

this case, were you familiar with 


17 

Public.Resource.Org? 


18 

MR. KAPLAN: Objection; vague. 


19 

THE WITNESS: I had heard of them. 


20 

BY MS. RUBEL: 


21 

Q In what context had you heard of them? 


22 

A I had probably met Carl Malamud at some 


23 

point. I don't recall meeting him, but I was aware 


24 

of his organization and that he had done different 


25 

things around making things accessible, and this is 
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1 

A 

I think we focused on the website that 


2 

didn't have an accessible sign-up process, and I'm 


3 

happy to 

find out which one of the three standards 


4 

bodies had that problem, just so I correctly testify 


5 

to that. 



6 

Q 

Sure. 


7 

A 

So I'm looking at my expert report. So we 


8 

focused our efforts on NFPA when we did our 


9 

in-person 

evaluation. 


10 

Q 

Is Rob Turner blind? 


11 

A 

Yes . 


12 

Q 

What is his background? 


13 


MR. KAPLAN: Objection; vague. 


14 


THE WITNESS: He's a blind engineer for my 


15 

nonprofit 

organization. 


16 

BY MS. RUBEL: 


17 

Q 

What -- what is his role -- 


18 


Is he employed by Benetech? 


19 

A 

Yes, he's employed by Benetech as a -- as 


20 

a Quality 

Assurance Engineer. 


21 

Q 

So what does he do in that role? 


22 

A 

He tests the quality of our products. 


23 

including 

our websites, evaluates accessibility, but 


24 

his focus 

is on our products. 


25 

Q 

Why did you seek Rob Turner's assistance? 
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1 A He's one of our blind employees who 

2 happens to be in the office regularly as opposed to 

3 being located in other locations; so I could go down 

4 and talk to him. 

5 Q So you asked -- 

6 You asked Rob to try to access standards 

7 from NFPA's website and see if he was able to do so? 

8 MR. KAPLAN: Objection; vague. 

9 Go ahead. 

10 THE WITNESS: Correct. First, I asked him 

11 to look at the sign-up process to see if he could 

12 sign up for a free reading account without needing 

13 assistance from a sighted person, and he wasn't able 

14 to do that. 

15 BY MS. RUBEL: 

16 Q Was there anything else you asked him to 

17 do? 

18 A After I pushed the "I Agree" button and 

19 got him through that, that roadblock, I also asked 

20 him to try to read the standard in question. 

21 Q Did you ask Rob to try to access any of 

22 the Plaintiffs' standards that are posted on Public 

23 Resource's website? 

24 A No, I did not. 

25 Q What are the Web content accessibility 
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MR. KAPLAN: It's okay. Got to give me a 

175 

2 

breath here. 


3 

THE WITNESS: Okay. I will. I'll try. 


4 

BY MS. RUBEL: 


5 

Q As part of your analysis, did you analyze 


6 

whether the tables in the HTML were accessible -- 


7 

the tables in HTML from Public Resource's website 


8 

were accessible to blind people? 


9 

A I don't recall evaluating tables in 


10 

detail. 


11 

Q Did you evaluate whether the graphics in 


12 

the standards -- in Plaintiffs' standards were 


13 

accessible to blind people from Public Resource's 


14 

website? 


15 

A No. 


16 

THE VIDEOGRAPHER: Is this a good place? 


17 

MS. RUBEL: Sure. We can take a break. 


18 

We ran out of tape. 


19 

THE VIDEOGRAPHER: This is end of Disk 2. 


20 

We're off the record at 3:37. 


21 

(Recess taken.) 


22 

THE VIDEOGRAPHER: This is beginning of 


23 

Disk 3. We're back on the record at 3:48. 


24 

BY MS. RUBEL: 


25 

Q Based on your analysis of the Plaintiffs' 
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1 standards that are available on Public Resource's 

2 website in HTML format, do you have enough 

3 information to determine whether an engineer who 

4 wanted access to that standard would be able to 

5 obtain all the necessary information from the 

6 standard? 

7 MR. KAPLAN: Objection; incomplete 

8 hypothetical. 

9 THE WITNESS: No. 

10 BY MS. RUBEL: 

11 Q Do you know how many of the standards on 

12 Public Resource's website -- how many of Plaintiff's 

13 standards that are available on Public Resource's 

14 website are only available in PDF format? 

15 A No. 

16 Q Do you know how many of the standards that 

17 are at issue in this case are only available on 

18 Public Resource's website in PDF format? 

19 A No. 

20 Q Do you know if the number that is 

21 available in PDF format is greater than the number 

22 available in HTML format? 

23 MR. KAPLAN: Objection; vague. 

24 THE WITNESS: It would be likely from my 

25 inspection of the directories that there would be 
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A 

(Reading): 


205 

2 


"Having reviewed the 



3 


accessibility of the same standards 



4 


content rendered by 



5 


Public.Resource.Org and those of 



6 


the free access options provided by 



7 


the NFPA, ASHRAE and ASTM, it is my 



8 


opinion that Public.Resource.Org 



9 


currently provides the only 



10 


accessible option for 



11 


people/citizens with print 



12 


disabilities to access these 



13 


standards." 



14 

Q 

And in forming that opinion, you compared 


15 

the standards that were available on 



16 

Public. 

Resource.Org's website with the free access 


17 

options 

provided by Plaintiffs in forming that 



18 

opinion 

; is that correct? 



19 

A 

Correct. 



20 

Q 

Did you evaluate any PDFs being sold 

by 


21 

NFPA in 

forming that opinion? 



22 

A 

No. 



23 

Q 

Did you evaluate any PDFs being sold 

by 


24 

ASHRAE 

in forming that opinion? 



25 

A 

No. 




(8 fiffd DEP ° 

www.CapitalReportrrr^ompany.com © 2015 



Case l:13-cv-01215-TSC r Qoci 


Page 21 of 87 


USCA ty et¥immi^ce p s mm^ 573 


1 

Q Did you evaluate any PDFs sold by ASTM in 

206 

2 

forming that opinion? 


3 

A No. 


4 

Q So when you say that Public.Resource.Org 


5 

currently provides the only accessible option for 


6 

people/citizens with print disabilities to access 


7 

these standards, you're excluding from that opinion 


8 

any PDFs that are being offered by the Plaintiffs? 


9 

MR. KAPLAN: Objection; misstates the 


10 

document and testimony, misleading and vague. 


11 

THE WITNESS: In that sentence I refer to 


12 

"free access options." 


13 

BY MR. REHN: 


14 

Q So when you said the only accessible 


15 

options, what you actually meant to say was the only 


16 

freely accessible options without charge? 


17 

MR. KAPLAN: Objection. Thane, you're 


18 

getting a little badgering here, but you can answer 


19 

the question. 


20 

THE WITNESS: Yes. 


21 

BY MR. REHN: 


22 

Q Is it possible that documents being sold 


23 

can be described as accessible to people with print 


24 

disabilities? 


25 

MR. KAPLAN: Objection; vague. 
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THE WITNESS: Yes. 

207 

2 

BY MR. REHN: 


3 

Q Have you ever described a PDF for sale as 


4 

being accessible to someone with print disabilities? 


5 

A Let's see. I'm not sure I've described -- 


6 

I'm sure I've described paid products in other 


7 

formats as accessible, and I'm sure that I've 


8 

described PDFs as accessible. I'm not sure I've 


9 

described a paid PDF product as accessible. 


10 

Q But it's fair to say that if the PDFs 


11 

being sold by Plaintiffs are accessible to people 


12 

with print disabilities, this sentence would be 


13 

potentially inaccurate? 


14 

MR. KAPLAN: Objection; misleading, vague. 


15 

argumentative. 


16 

THE WITNESS: I think it's accurate as I 


17 

wrote it, but I'm happy, as we have, to clarify what 


18 

I meant by that sentence. 


19 

BY MR. REHN: 


20 

Q Sure. 


21 

And to clarify what you meant was you were 


22 

comparing just the free access options provided by 


23 

the Plaintiffs on their websites with the content 


24 

rendered by Public.Resource.Org? 


25 

A Those were the content that I evaluated. 
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1 

yes . 


208 

2 

Q 

Are you aware that NFPA sells eBook 


3 

versions 

of some of its standards? 


4 

A 

I'm -- I'm not sure I'm aware of that. 


5 

Q 

And did you evaluate any eBook versions of 


6 

standards 

sold by NFPA or the other two Plaintiffs 


7 

in this case? 


8 

A 

No. I was not asked to. 


9 

Q 

So do you have an opinion on whether the 


10 

eBook versions sold by NFPA and the other two 


11 

Plaintiffs are accessible to persons with print 


12 

disabilities? 


13 

A 

I can't express an opinion without looking 


14 

at them. 



15 

Q 

Do you have opinion on whether the PDFs 


16 

being sold by NFPA and the other two Plaintiffs in 


17 

this case 

are accessible to persons with print 


18 

disabilities ? 


19 

A 

No. But in my experience, most PDFs are 


20 

not accessible or are not — let me correct that. 


21 

Not as accessible as, say, HTML versions of those 


22 

would be 

since accessibility is on a spectrum. 


23 

Q 

But you don't have an opinion as to these 


24 

specific 

PDFs that are sold by NFPA? 


25 

A 

I haven't examined them. 
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MR. KAPLAN: Do you want to take a break? 

209 

2 

THE WITNESS: No, I'll keep going. I'll 


3 

get water next time. 


4 

MR. REHN: You can take a break whenever 


5 

you want. 


6 

THE WITNESS: No problem. 


7 

BY MR. REHN: 


8 

Q So you mentioned earlier that you were 


9 

aware of an NFPA standard that was available on 


10 

Bookshare? 


11 

A Uh-huh. 


12 

Q If I could just show you a document. 


13 

MR. REHN: What number are we up to? 


14 

THE REPORTER: We're at 4002. 


15 

(Plaintiffs' Exhibit 4002 marked 


16 

for identification.) 


17 

BY MR. REHN: 


18 

Q Do you recognize this as a printout of the 


19 

Bookshare Web page? 


20 

A Yes. 


21 

Q And you'll see here that this says "NFPA 


22 

70-2014 Electrical Code Book"? 


23 

A Yes. 


24 

Q And I believe you said earlier that your 


25 

understanding was that a partner of Bookshare had 
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submitted this to Bookshare? 

210 

2 

A 

Correct. 


3 

Q 

Is that on the second page of this 


4 

exhibit, 

you see it says "Submitted by Daproim 


5 

Africa"? 



6 

A 

Yes . 


7 

Q 

Is that who you understand submitted this 


8 

document 

9 


9 

A 

Yes . 


10 

Q 

How does Bookshare enable certain persons 


11 

to share 

documents with Bookshare? 


12 

A 

So we're talking about essentially the 


13 

content 

intake mechanisms at Benetech, and you'd 


14 

like me 

to enumerate those different mechanisms? 


15 

Q 

Well, let's start with this Daproim 


16 

Africa. 

They're an adult educator; is that what you 


17 

said earlier? 


18 

A 

No. 


19 

Q 

What's your understanding -- 


20 


Do you know who this submitter is? 


21 

A 

Yes . 


22 

Q 

And who are they? 


23 

A 

They're a subcontractor to Benetech for 


24 

books -- 

they do the proofreading services on books 


25 

we've been asked for by a student. 
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1 

Q So a student would have submitted a 

211 

2 

request and because it was educational, you would 


3 

have approved that request; is that what you 


4 

testified to earlier? 


5 

MR. KAPLAN: Objection; misstates 


6 

testimony, calls for speculation. 


7 

THE WITNESS: Yes. 


8 

BY MR. REHN: 


9 

Q And after you approved that request, you 


10 

would have had a subcontractor proofread the 


11 

document and then upload it to Bookshare? 


12 

MR. KAPLAN: Objection; vague, calls for 


13 

speculation. 


14 

THE WITNESS: Yes. 


15 

BY MR. REHN: 


16 

Q Do you allow anybody to submit documents. 


17 

other than subcontractors? 


18 

A Yes. 


19 

Q Do you engage in any quality control on 


20 

documents that persons, other than subcontractors. 


21 

submit? 


22 

A Yes. 


23 

Q And what is that quality control process? 


24 

A We run an automated quality evaluator that 


25 

scores the document on, for example, looking for OCR 
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errors, looking for swear words, looking for length; 

212 

2 

it's supposed to be 100-page document and it's a 


3 

1-page document that's submitted or a thousand-page 


4 

document. 


5 

And we also have a human being take a 


6 

brief look at the document, kind of just do a check 


7 

also to make sure that it makes -- that it makes 


8 

sense. 


9 

Q Are those documents that are submitted 


10 

generally PDFs or Word documents or HTML? What 


11 

format do they usually come in? 


12 

MR. KAPLAN: Objection; lacks foundation. 


13 

calls for speculation, vague. 


14 

THE WITNESS: Bookshare has a detailed 


15 

description of how we want documents submitted to 


16 

us. In general, we prefer documents that are -- 


17 

when they're coming from volunteers that are 


18 

scanning that are more Microsoft Word or RTF, which 


19 

is a related format. For those books that are 


20 

scanned, that's our preferred format, and we 


21 

wouldn't accept a PDF. 


22 

BY MR. REHN: 


23 

Q So you accept documents or books that are 


24 

scanned by volunteers? 


25 

MR. KAPLAN: Objection; misstates 
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1 

testimony. 

213 

2 

THE WITNESS: We accept books from quite a 


3 

number of sources of which volunteers is one source. 


4 

BY MR. REHN: 


5 

Q Can anybody sign up to be a volunteer to 


6 

submit books or documents? 


7 

A I think right now you have to be a U.S. 


8 

resident or organization. 


9 

Q So any U.S. resident could sign up to 


10 

submit books or documents to Bookshare? 


11 

A Yes, and they must agree to our volunteer 


12 

agreement that specifies the limitations on what 


13 

they can do and what they can't do. 


14 

Q So if a concerned citizen wanted to get 


15 

books or documents accessible to the visually 


16 

impaired, they could volunteer to scan those 


17 

documents, proofread them, ensure that they're free 


18 

of errors and then submit them to Bookshare, and if 


19 

they passed your quality control process, you would 


20 

make them available on your website? 


21 

MR. KAPLAN: Objection; incomplete 


22 

hypothetical, vague. 


23 

THE WITNESS: Yes. 


24 

BY MR. REHN: 


25 

Q Do you try to encourage volunteers to 
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1 

and implementing the standard? 

218 

2 

A No. 


3 

Q Did you see any graphical material in that 


4 

standard? 


5 

A I recall seeing graphical material in the 


6 

standards I evaluated. 


7 

Q Did you assess whether that graphical 


8 

material was accessible via a screen reader in the 


9 

HTML version of the Public Resource website? 


10 

MR. KAPLAN: Objection; vague. 


11 

THE WITNESS: I didn't check for 


12 

additional accessible metadata on the images. 


13 

BY MR. REHN: 


14 

Q So do you have an opinion on -- 


15 

Do you have enough information to know 


16 

whether a visually impaired fire safety professional 


17 

could use the HTML version of NFPA 101 that is 


18 

available on Public Resource's website and safely 


19 

rely on that for professional purposes? 


20 

MR. KAPLAN: Objection; vague. 


21 

THE WITNESS: I am not a fire professional 


22 

expert, so I can't evaluate how this applies 


23 

specifically to that profession. 


24 

BY MR. REHN: 


25 

Q So the answer to my question is "no"? 
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1 



FURTHER EXAMINATION 

256 

2 

BY 

MR. 

REHN: 


3 


Q 

Good evening, Mr. Fruchterman. 


4 


A 

Hello. 


5 


Q 

And do you understand that you're still 


6 

under oath? 


7 


A 

Yes . 


8 


Q 

Has anything happened between now and the 


9 

last time we spoke that would affect your ability to 


10 

answer 

my questions fully and truthfully? 


11 


A 

No. 


12 


Q 

So I'd like to direct your attention to an 


13 

exhibit 

that we are marking as Exhibit Number 4006. 


14 


A 

Yes . 


15 



(Plaintiffs' Exhibit 4006 to be 


16 



marked for identification.) 


17 

BY 

MR. 

REHN: 


18 


Q 

Do you recognize this as an e-mail to 


19 

yourself from Rob Turner that was sent on April 10th 


20 

of 

this 

year at 10:56 a.m.? 


21 


A 

Yes . 


22 


Q 

And the subject line is "OCR Document"? 


23 


A 

Yes . 


24 


Q 

Do you recall receiving this e-mail? 


25 


A 

Yes . 
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1 

Q And do you know what OCR document 

257 

2 

Mr. Turner is referring to in the subject line? 


3 

A Is there, like, an immediately prior 


4 

document that actually mentions this? Sorry. 


5 

Sorry. I mean, can I look through the list of 


6 

produced documents? 


7 

MR. KAPLAN: You just got to answer his 


8 

question. 


9 

BY MR. REHN: 


10 

Q Based on this e-mail, do you know which 


11 

document he's referring to? 


12 

A I don't remember which one of the 


13 

image-based standards I shared with him, no. But it 


14 

was one of the image-based PDFs that I asked him to 


15 

look at. 


16 

Q So the image-based PDFs that you sent 


17 

Mr. Turner were -- those were PDFs you had taken 


18 

from Public Resource's website; is that correct? 


19 

MR. KAPLAN: Objection; argumentative. 


20 

misleading and vague. 


21 

THE WITNESS: It probably was an 


22 

image-based PDF from the Public.Resource.Org 


23 

website, and that's my -- that's my recollection. 


24 

Yes . 


25 
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BY MR. REHN: 


258 

2 

Q 

Did you make any image-based PDFs of 



3 

documents 

from any of Plaintiffs' websites? 



4 

A 

I didn't make any documents from 



5 

Plaintiffs' websites. I downloaded whatever 



6 

document 

-- no, I downloaded -- I viewed the 



7 

document. 

yes. So, no. 



8 

Q 

After you sent him a document, it would 


9 

have been 

one from Public Resource's website? 



10 

A 

That's correct. Thank you. 



11 

Q 

And if I could direct you to the last 


12 

sentence 

of the first paragraph of his e-mail. 

would 


13 

you read 

that sentence, please? 



14 

A 

The one "I don't think..."? 



15 

Q 

Yes . 



16 

A 

Yes . 



17 


"I don't think this type of 



18 


document can be considered to be 



19 


accessible." 



20 

Q 

So based on your prior testimony, is 

it 


21 

your understanding that he is saying that the 



22 

image-based PDF from Public Resource's website 

that 


23 

you sent 

to Mr. Turner, in his opinion, cannot 

be 


24 

considered to be accessible? 



25 


MR. KAPLAN: Objection; misleading. 
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1 

argumentative, vague. 

259 

2 

THE WITNESS: I think Rob Turner doesn't 


3 

believe it meets our accessibility standards, which 


4 

is what his job is to primarily work on our library 


5 

for the blind. We would not post an image-based PDF 


6 

and call it accessible. 


7 

BY MR. REHN: 


8 

Q And do you agree with Mr. Turner's 


9 

assessment that this type of document cannot be 


10 

considered to be accessible? 


11 

A I think it's less accessible than many of 


12 

the other documents and more than others, as I wrote 


13 

in my expert report. I can probably quote from the 


14 

report. 


15 

Q There's no question pending. So... 


16 

A Okay. I would direct you to my last 


17 

sentence of my report -- 


18 

MR. KAPLAN: Jim, there's no question 


19 

pending. 


20 

THE WITNESS: All right. 


21 

MR. REHN: I have no further questions. 


22 

And I believe that concludes Plaintiffs' questioning 


23 

of this witness. 


24 

MR. KAPLAN: I have no questions at this 


25 

time. 
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their codes, and I wouldn't be able to tell you what 12:31:09 

codes, but I believe it goes all the way back to 2004, 12:31:11 

2005. 12:31:18 

Q. In the answer you just gave, you referred to 12:31:19 

when you started working with NFPA and exchanged 12:31:23 

information with them. When do you date that? 12:31:23 

A. That would be, I think I've met — the 12:31:25 

standards community in Washington is a small 12:31:31 

community. So I've met the various Washington 12:31:34 

representatives for agencies. Excuse me. For SDO's, 12:31:35 

standards development organizations, many times in my 12:31:39 

career. And I would say I've worked cooperatively and 12:31:41 

individually whenever necessary throughout my career 12:31:45 

at ASTM. So... 12:31:48 

Q. Well, I think that doesn't quite answer my 12:31:56 

question. I think you said you developed this 12:31:59 

interest when you began to hear -- sorry. When you 12:32:02 

began to — when you started working with them on 12:32:06 

exchanging information. I'm just trying to find out 12:32:10 

what year you're referring to when you said that. 12:32:12 

MR. FEE: Objection. Mischaracterizes his 12:32:14 

testimony. 12:32:15 

THE WITNESS: I wouldn't be able to give you 12:32:19 

an exact year except for I know when we began the APCO 12:32:20 

related work, that was 2011 time frame. 12:32:25 
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BY MR. BRIDGES: 12:32:29 

Q. And did your interest in providing a reading 12:32:31 

room arise about the same time as the APCO engagement 12:32:32 

arose? 12:32:38 

A. Similar time line. I believe it began to -- 12:32:40 

I began to introduce the idea and socialize it before 12:32:43 

then. Maybe a year or so before then. 12:32:45 

Q. You introduced the idea of a reading room? 12:32:48 

A. The idea of figuring out a way to strike the 12:32:51 

right balance. I think another idea we had at the 12:32:53 

time that I introduced was perhaps figuring out if 12:32:57 

there was a way we could provide better summaries of 12:33:01 

our standards to the public rather than relying on 12:33:04 

abstracts. So there was various ideas that I began to 12:33:07 
socialize with ASTM staff about how to strike this 12:33:13 

delicate balance between providing the public with 12:33:17 

greater access to our documents while still preserving 12:33:20 
what we need to preserve in order to meet — continue 12:33:25 

the enterprise of developing standards, keeping the 12:33:28 

barriers to participation low, and ensuring that would 12:33:31 
continue to provide the important value that we do in 12:33:35 

high-quality market-relevant standards that protect 12:33:39 

the public. 12:33:42 

Q. How did you introduce the idea of providing a 12:33:44 

reading room in the discussion you were referring to? 12:33:46 
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specifically. Using the NIST database as a guideline, 12:49:53 
we've incorporated, you know, as much of that as 12:50:02 

possible in the reading room. At times I believe we 12:50:04 

also tried to add a little bit more intelligence to it 12:50:06 
to determine if an agency was undertaking a subsequent 12:50:09 
rule-making, and we became aware that the agency had 12:50:18 

published a new final rule which either changed the 12:50:24 

reference to an ASTM standard that we had placed in 12:50:27 

the reading room or added a new ASTM standard to the 12:50:31 

reading room. 12:50:38 

Then we took steps to add that to the reading 12:50:39 
room. It's not an exact science. We don't pay a 12:50:42 

vendor to perform the service for us. We rely either 12:50:48 

exclusively on the NIST database or we -- it's based 12:50:55 

on intelligence that we've gathered about new 12:50:58 

rulemakings. 12:51:01 

Q. How do you gather intelligence about 12:51:03 

incorporations of ASTM standards by reference? 12:51:08 

A. Well, as much as possible we read the federal 12:51:14 

register. I'd like to think we read it on a regular 12:51:17 

basis, but sometimes it's more infrequent than that. 12:51:20 

So we will search key terms in the federal register to 12:51:24 

see if it's mentioning ASTM and if there's a rule that 12:51:30 

has resulted in the publication of standards. And 12:51:34 

sometimes we're ahead of it because ASTM has a policy 12:51:38 
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of working with agencies during the notice of proposed 12:51:41 
rule-making process. 12:51:45 

Any agency that comes to us and asks us to 12:51:46 

put a standard up for public review during the public 12:51:50 

review period of a rule, we work with them to make 12:51:53 

that possible. So at times we know that a certain 12:51:57 

number of ASTM standards have been in a notice to 12:52:01 

proposed rulemaking and that the new rule's expected 12:52:04 

to come out, so we can look for it. 12:52:08 

Q. Does ASTM provide assistance to the 12:52:16 

government in any way when the government is 12:52:18 

considering whether to incorporate an ASTM standard by 12:52:20 

reference? 12:52:23 

MR. FEE: Objection. Vague. 12:52:24 

THE WITNESS: So we do -- I'm familiar with a 12:52:29 
couple things that either I do or a member of my staff 12:52:31 

does. We look to see -- when we're aware that an ASTM 12:52:34 

standard is going to be used and incorporated by 12:52:39 

reference in some type of an action, we look to see 12:52:43 

what version of the standard and what designation of 12:52:46 

the standard is being used, and I believe on occasion 12:52:50 

if they're using -- proposing to use an outdated 12:52:54 

version of a standard, or, quite frankly, we've seen 12:52:59 

errors where they've attempted to use an ASTM biofuel 12:53:02 

standard, and rather than referencing D6751 they've 12:53:06 
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A. I believe close to the full collection. So 

15:31:36 

as many as 1,300 ASTM documents. 

15:31:38 

Q. What announcements to the press did ASTM make 

15:31:43 

about its reading room going live? 

15:31:48 

A. I don't recall if we made a lot of 

15:31:54 

announcements when it went live in January. I believe 

15:31:55 

we were concerned about if it would function and work. 

15:31:58 

and I think we wanted to get a little experience with 

15:32:05 

it before we broadcast it too widely. 

15:32:08 

Q. Did ASTM ever make announcements to the press 

15:32:12 

about the availability of its reading room? 

15:32:15 

MR. FEE: Objection. Vague. 

15:32:17 

THE WITNESS: Yes. Through our flagship 

15:32:19 

communication, Standardization News, which we 

15:32:21 

delivered to all of our members and stakeholders. I 

15:32:25 

believe 30,000 individuals receive it six times a 

15:32:27 

year. Mention of it was made in the magazine. 

15:32:33 

BY MR. BRIDGES: 

15:32:36 

Q. When was that? 

15:32:36 

A. I'm sorry. I don't know specifically. 

15:32:37 

Q. How long after the launch of the reading room 

15:32:39 

did that occur? 

15:32:45 

A. I'm sorry. I don't recall. It was in 2013. 

15:32:53 

Q. Did ASTM ever make an announcement to the 

15:32:59 

press about the availability of its reading room 

15:33:03 
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beyond the announcement in Standardization News? 

MR. FEE: Objection to form. 

THE WITNESS: I believe it was also announced 
at the ASTM annual business meeting in 2013. 

BY MR. BRIDGES: 

Q. The "ASTM business meeting" being a meeting 
of ASTM members and stakeholders? 

A. Yes. 

Q. What other public announcements did ASTM make 
about the availability of its reading room beyond 
announcements to its own members and stakeholders? 

A. I also believe that there was a reference to 
it in the ASTM annual report in 2013, which was 
published in 2014. 

Q. What else? 

A. I make it part of my message, when I'm 
visiting with stakeholders that I interact with, that 
ASTM has this reading room. 

Q. What else? 

A. Jim Thomas, our president, mentions it in his 
interactions on a worldwide basis. 

Q. With whom? 

A. Jim Thomas is a popular figure in the 
standards community, a well-known expert, and he 
speaks to many groups. So I wouldn't be able to give 


15:33:08 
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you specifics without reviewing his calendar. 

Q. What else? 

A. ASTM has an electronic newsletter. I believe 
we mentioned it in the newsletter in 2013. 

Q. To ASTM's members and stakeholders? 

A. Yes. To anyone interested in subscribing. 

Q. What else? 

A. We previously discussed some efforts to 
educate policy makers and stakeholders in Washington 
through an APCO public relations campaign. I believe 
the reading room was part of that messaging as well in 
2013. 

Q. What else? 

A. That's all I can recall at this time. It had 
a place on our website as well. 

Q. Of all the persons who had access to — 
sorry. Were you about to mention another? 

A. I'm sorry. We also sent a few letters to 
agencies informing them of the creation of the reading 
room. 

Q. By "agencies," do you mean government 
agencies? 

A. To government agencies, to the office of 
management and budget, and to the office of the 
federal register at NARA, the National Archives 
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A. 

Yeah, about 12 years ago I had one taken. 

Q. 

Is that the only deposition? 

A. 

That's the only one. 

Q. 

What kind of case did that involve? 

A. 

That was a personnel matter for our 


organization 


Q. 

Did you testify at trial? 

A. 

No, I did not. 

Q. 

Did you have a chance to meet with 

Mr. Lewis 

or other counsel before this deposition to 


prepare for the deposition? 


A. 

Yes, I did. 

Q. 

I'll ask you to look at Exhibit 1076 — 

(Defendant's Exhibit 1076 was marked for 


identification 


Q. 

(By Mr. Bridges) — which is Defendant's 

Notice of 

30(b)(6) deposition of ASHRAE. Please take 

a look at 

it, Mr. Comstock. 

Do you understand that you are here today 


testifying as a representative of ASHRAE on Topics 4 
5, 7, 8, 9, 10, 12, 13, 14, 18, 23, 24, 30, and 31? 


A. 

Yes, that's my understanding. 

Q. 

When did ASHRAE start providing a reading 


room for public access to ASHRAE's standards? 


A. 

We made selected standards available for 
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read-only access, and I believe that was about 15 

years ago. I don't have the exact date. It was in 

that — that range of time. 

Q. How did ASHRAE select what standards to 
make available? 

A. These are our -- our most popular 
standards, the ones for which there was the greatest 
demand. 

Q. How many standards — strike that. 

How many current standards does ASHRAE 

publish? 

A. I don't have the exact number. My 
recollection would be in the neighborhood of — of 
75 . 

Q. How many of those standards are on ASHRAE's 
reading room available to the public now? 

A. At the current time, I believe there are 10 
of those standards available. 

Q. Does ASHRAE also make available through its 
reading room earlier versions of those 10 standards? 

A. We provide — we provide the current 
versions of those standards. 

Q. But not the earlier versions? 

A. I believe that's the case. 

Q. Do you know why ASHRAE began providing 

Page 11 

Veritext Legal Solutions 
866 299-5127 

JA1996 



1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 


Case l:13-cv-01215-TSC Document 155-8 Filed 01/21/16 Page 61 of 87 
Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 238 of 573 

public access to some of its standards? 

A. We were actually hoping to increase our 
sales of those standards. It would be to the -- to 
allow somebody to view those standards, but not be 
able to download those standards or print those 
standards. So that would drive demand for those -- 
for those standards. 

Q. What was ASHRAE's experience in that 
regard? 

A. It was — our experience was that it was 
relatively flat. It didn't have -- seem to have much 
of a positive impact, nor in — in that case did it 
seem to have a negative impact. 

Q. Does ASHRAE have information about how many 
persons have accessed the standards in its reading 
room? 

A. We did. We changed the — the — the 
software platform from which they were made available 
for viewing. We originally used — we originally 
used a RealRead vendor-supplied system and then we 
went -- they went out of business, I believe, and 
then we switched to iWrapper. 

But I -- I know for certain when we were 
with RealRead, we would track the views. There was 
no registration so we wouldn't know who those people 
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So I — I would — I would assume that 
the — the largest — the most substantial revenue 
stream that they provide to us in royalty comes from 
network licenses. 

Q. And how much would you estimate that to be 
on an annual basis? 

A. Do you mean the — the -- the total revenue 
or the part from — or the part from network 
licenses ? 

Q. Let's say the total revenue from 
value-added resellers to begin with and then 
understanding whether you can break out network — 

A. Yeah. 

Q. — licenses. 

A. Our -- our total royalty revenue would be 
roughly 1.2 million to 1.4 million. 

Q. And when you identify your total royalty 
revenue, that revenue number is separate from the 
revenue number you gave me earlier about publications 
revenue; is that correct? 

A. Yes, that's correct. 

Q. So to understand the total — I hate to use 
the word, but monetization value of publications, one 
would have to add in the publications revenue and the 
royalty revenue; correct? 
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A. That is correct. 

Q. What other components would be missing if I 
had just the publication revenue and the royalty 
revenue? 

A. Now, we are speaking just — of just 
publications ? 

Q. Right, and really specifically standards. 

A. Standards. Just running through our 
financial statements in my mind. That — that's it. 

Again, there's educational components that 
we may use standards in which — but there's no — 
but sometimes like we include a standard in a 
registration fee for a conference, so there's no 
direct revenue from that standard. 

But if you added together the royalty sales 
and you added together our direct sales of 
publications, that would represent our — our total 
publication revenue. 

Q. Do you have an estimate as to what 
percentage of that total revenue is attributable, in 
your mind — or in ASHRAE's mind, to all versions of 
90.1? 

MR. LEWIS: Objection. 

THE WITNESS: So what percentage of our 

total publications revenue, if that total 
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revenue is both what we 

sell and the royalties. 

what 

s the component of 

that that is 


attributable to — 



Q. 

(By Mr. Bridges) 

90.1 — 


A. 

— 90.1? 



Q. 

— all versions. 



A. 

Yeah. And let me 

just go through 

some math 

as I'm-- 

as I'm speaking. 




And this would not be any of the kind of 


indirect educational or, you know, credibility and 
other — other ways that that may impact us. 


Q. 

Right. 



A. 

Yes, just give me - 

- okay. 

Now let me just 


run through those numbers now. 

Well, when it gets to the royal- — the 
problem is for the royalty part I'm really making 
guesses, because it's -- because I don't have — you 
know, it -- it — I -- I don't have those numbers, 
you know, broken down as such. 

Q. I'll just ask you for your best estimate. 
A. Best estimate. 

MR. LEWIS: Objection. 

THE WITNESS: So the best estimate, if the 
total was $450,000 — 

Q. (By Mr. Bridges) Out of the total. 
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A. Out of the total as an estimate, just 
conjecturing. 

Q. Is — excuse me, I may have — I don't 
think I asked the exact same question. I may have 
asked a similar question earlier. Forgive me if I 
repeat myself because I'm working on one hour of 
sleep. 

Is 90.1 ASHRAE's — I think — strike that. 

I think you said it was ASHRAE's most 
popular standard; is that correct? 

A. (Witness nodded head affirmatively.) 

MR. LEWIS: Objection. 

Q. (By Mr. Bridges) What would you consider 
the second most popular standard to be? 

A. Second I would consider Standard 62.1, 
which is ventilation requirements for buildings. 

Q. What would round out the rest of the top 
five, in your view? 

A. Top five. Standard 55, which is a thermal 
comfort standard; Standards 15 and 34, which relate 
to refrigerant use and -- in air-conditioning and 
refrigeration systems. 

Q. I think, based on the number of years 
you've been at ASHRAE, is it correct that you started 
at ASHRAE before ASHRAE first published 90.1? 
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context. You nodded, but the court reporter can't 

take nods down. 

Do you understand, broadly speaking, 
monetization of publications through revenue sources 
like purchasing and licensing and the like? 

A. Yes. 

Q. And royalties? 

A. Yes. 

Q. What proportion of ASHRAE's yearly revenues 
comes from the monetization of its standards as 
publications ? 

A. I'm making sure I'm doing the math right. 

Q. That's fair. That's fair. 

A. Let's see. It would be — directly 
attributable to standards would be approximately 
10 percent. 

Q. How else does ASHRAE earn revenue, other 
than through the monetization of its standards? 

A. Membership dues, conference registrations, 
advertising, subscription sales, educational course 
registrations, certification, exposition income. 

And when you said "publications," if — so 
in addition to publications, we have books. So 
books, if — if — if -- if that's -- if you 
distinguish between standards in your questions, then 
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there would be books. And I believe that -- that — 

that covers it. 

Q. Roughly what percentage of ASHRAE's 
expenses pertain to the organization and supervision 
of the standards development process and the costs of 
publication and the costs of administering the 
permissions and distributions and the like? 

MR. LEWIS: Objection. 

THE WITNESS: I can speak to the side of 
that process that deals with the — the — the 
publications part. I do not know what the — 
the costs would be to support the development of 
the product. My role begins when we push that 
standard out to the -- to -- to the marketplace. 

What would be — I — I'd probably say 
there are staff salaries that would be 
attributable to standards activities from the 
publication side of things, production, so on. 

If you add portions of people's time together, 
we're probably speaking of four people from the 
publications side. 

And then the — the cost of the 
infrastructure for the book- — for the 
bookstore, the on-line process, and warehousing, 
and finally the — the — the work that may be 
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what a subvention is of a publication? 

A. I do not. 

Q. Has ASHRAE ever received any grants to 
support the publication of any particular standards? 

A. I have no knowledge of ASHRAE receiving 
funds for that. 

Q. Is ASHRAE aware of any monetary losses that 
it has suffered as a consequence of the defendant's 
conduct in this case? 

A. I can't speak to any -- any tracking of — 
of losses. And anecdotally, people say if — they've 
asked me if a standard is available on the Internet, 
is that — is that allowed, is that permissible, so 
we -- in those cases, we will seek to remove them. 

But we don't -- we — I don't have any 
record of tracking such loss of — of revenue. 

Q. Apart from tracking it, does ASHRAE have 
any information regarding monetary losses it has 
suffered as a consequence of defendant's conduct? 

A. I -- I do recall there was one message we 
got from somebody who refer- -- I think it was 
somebody with Carrier Corporation, if I recall, who 
referred to -- who referred to that. I don't know if 
they had intended to purchase or not, but that was 
one specific case I do recall. 
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IN THE UNITED STATES DISTRICT COURT 
FOR THE DISTRICT OF COLUMBIA 


AMERICAN SOCIETY FOR TESTING AND Case No. l:13-cv-01215-TSC-DAR 
MATERIALS d/b/a ASTM INTERNATIONAL; 

NATIONAL FIRE PROTECTION 

ASSOCIATION, INC.; and Action Filed: August 6, 2013 

AMERICAN SOCIETY OF HEATING, 

REFRIGERATING, AND AIR 
CONDITIONING ENGINEERS, 

Plaintiffs-Counterdefendants, 


PUBLIC.RESOURCE.ORG, INC., 

Defendant-Counterclaimant. 


SUPPLEMENTAL DECLARATION OF CARL MALAMUD IN FURTHER SUPPORT 
OF DEFENDANT’S MOTION FOR SUMMARY JUDGMENT 

I, Carl Malamud, declare as follows: 

1. Iam over the age of 18 years and am fully competent to testify to the matters stated in this 
declaration. 

2. This declaration is based on my personal knowledge. If called to do so, I would and could 
testify to the matters stated herein. 

3. I am the President and sole employee of Public.Resource.Org, Inc. (“Public Resource”), 
which is a 501(c)(3) non-profit corporation headquartered in Sebastopol, California. I have 
worked at Public Resource since I founded the organization in 2007. It is my only source of 
employment. 
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4. Public Resource does not charge for access to the archive of laws and other government 
authored materials on the domains under the public.resource.org website. 

5. In August of 2008,1 posted the Alabama Electrical Code, Arkansas Electrical Code, Idaho 
Electrical Code, Minnesota Electrical Code, North Carolina Electrical Code, North Dakota 
Electrical Code, New Hampshire Electrical Code, Rhode Island Electrical Code, South 
Dakota Electrical Code, Utah Electrical Code, and Wyoming Electrical Code to the website 
law.resource.org. 

6. Each of these codes contains the unmodified text of the 2008 National Electrical Code. 

7. For instance, the Alabama Electrical Code, Alabama Building Commission Administrative 
Code chapter 170-X-2, consists of the complete, unmodified 2008 National Electrical Code 
with five pages of Alabama-specific material placed at the front of the document. Attached 
to Public Resource’s Index of Consolidated Exhibits as Exhibit 16 is a true and correct 
copy of the first 34 pages of the Alabama Electrical Code that I posted on the Public 
Resource website in August of 2008. Because the entire document is over 800 pages I have 
not included it in its entirety, and Public Resource will have the entire document available 
at the hearing on Public Resource’s Motion for Summary Judgment and requests permission 
to lodge it or file it formally at the hearing, instead of putting such a large document on 
PACER. The full document will be made available for Plaintiffs to inspect at their request. 

I declare under penalty of perjury under the laws of the United States that the foregoing is true 

and correct. 

Executed this 4th day of February, 2016 at Sebastopol, California. 



Carl Malamud 
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EXECUTIVE OFFICE OF THE PRESIDENT 
Office of Management and Budget 

OMB Circular A-l 19: Federal Participation in the Development and Use of Voluntary Consensus 
Standards and in Conformity Assessment Activities 

AGENCY: Office of Management and Budget, Executive Office of the President 
ACTION: Final Revision of OMB Circular A-l 19 

SUMMARY: The Office of Management and Budget (OMB) has revised Circular A-l 19, "Federal 
Participation in the Development and Use of Voluntary Consensus Standards and in Conformity 
Assessment Activities" (hereinafter, Circular A-l 19, or, the Circular) in light of developments in 
regulation, standards, and conformity assessment since the Circular was last revised in 1998. 

DATES: Effective upon publication 

ADDRESSES: Direct any comments or inquiries to the Office of Information and Regulatory 
Affairs, Office of Management and Budget at CircularA-119@omb.eop.gov . 

SUPPLEMENTARY INFORMATION: 

I. Existing OMB Circular A-119 

II. Discussion and Responses to Significant Comments 

III. Section-by-Section Discussion 

Existing OMB Circular A-119. The vibrancy and effectiveness of the U.S. standards system in 
enabling innovation depends on continued private sector leadership and engagement. Our 
approach—reliance on private sector leadership, supplemented by Federal government contributions 
to discrete standardization processes as outlined in OMB Circular A-119—remains the primary 
strategy for government engagement in standards development. 

The policies of Circular A-119 are intended to encourage Federal agencies to benefit from the 
expertise of the private sector, promote Federal agency participation in standards bodies to support 
the creation of standards that are useable by Federal agencies, and minimize reliance on 
government-unique standards where an existing standard would meet the Federal government’s 
objective. 

The National Technology Transfer and Advancement Act of 1995 (Public Law 104-113; hereinafter 
known as “the NTTAA”) codified pre-existing policies on the development and use of voluntary 
consensus standards in OMB Circular A-119, established additional reporting requirements for 
agencies, and authorized the Department of Commerce’s National Institute of Standards and 
Technology (NIST) to coordinate conformity assessment activities. 

Revision of OMB Circular A-119. OMB is revising this Circular in light of developments in 
regulation, standards, and conformity assessment since the Circular was last revised in 1998. These 
revisions reflect the experience gained by U.S. agencies in implementing the Circular since 1998, 
and concluding and implementing U.S. trade agreements, as well as developments in domestic and 
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international regulatory, standards, and conformity assessment policies. 

The revised Circular reflects and supports the regulatory policies and principles set out in relevant 
executive orders. OMB notes, in particular, the requirements of four executive orders, three of 
which were issued after 1998: 

• Executive Order 12866 (“Regulatory Planning and Review”) states that regulations must be 
consistent with law; regulations must identify the nature and significance of the problem; 
agencies must identify and assess alternatives to address the problem along with the costs 
and benefits of each alternative; and the approach selected should maximize net benefits to 
society; 

• Executive Order 13563 (“Improving Regulation and Regulatory Review”) emphasizes that 
the U.S. regulatory system “must protect public health, welfare, safety, and [the] 
enviro nm ent while promoting economic growth, innovation, competitiveness, and job 
creation,” and stresses the importance of public participation and careful consideration of 
both benefits and costs; 

• Executive Order 13609 (“Promoting International Regulatory Cooperation”) directs Federal 
agencies to better coordinate U.S. priorities and positions with respect to international 
regulatory cooperation efforts across U.S. Federal agencies. This includes promoting good 
regulatory practices both in the United States and internationally, as appropriate, and 
considering reforms that address unnecessary differences in regulatory requirements 
between the United States and its major trading partners; and 

• Executive Order 13610 (“Identifying and Reducing Regulatory Burdens”) institutionalizes 
the retrospective review mechanism set out in Executive Order 13563 and calls on agencies 
to reduce the cumulative effects, including the cumulative burdens, of regulation. 

In addition to the above Executive Orders, on January 17, 2012, OMB’s Office of In formation and 
Regulatory Affairs (OIRA), the Office of Science and Technology Policy (OSTP), and the Office of 
the United States Trade Representative (USTR) released a Memorandum to Federal agencies on 
“Principles for Federal Engagement in Standards Activities to Address National Priorities” (see 
http://www.whitehouse.gov/sites/default/files/omb/memoranda/2012/m-12-08) . 

The revisions to Circular A-l 19 inform agencies of their statutory obligations in standards-setting 
activities. Specifically, 19 U.S.C. § 2532 provides that “[n]o Federal agency may engage in any 
standards-related activity that creates unnecessary obstacles to the foreign commerce of the United 
States” and specifies four requirements for agencies’ standards-related activities: 

1. Ensuring that imported products are not treated less favorably than like domestic products or 
imported products from other countries; 

2. Taking into consideration international standards - and basing standards upon them if 
appropriate; 

3. Developing standards based on performance criteria rather than design criteria when 
appropriate; and 

4. Ensuring foreign suppliers have access to conformity assessment procedures on the same 
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basis as other domestic and foreign suppliers of like products. 

In order to gather relevant information for this revision, OMB published a Request for 
In formation (RFI) in the Federal Register on March 30, 2012 (77 Fed. Reg. 19357) on “whether and 
howto supplement Circular A-119.” The notice also announced OMB would be holding a public 
workshop at NIST on May 15, 2012. 

The RFI and public workshop allowed interested stakeholders to provide valuable input to OMB, 
NIST, Federal regulators, and other relevant agencies on how the Federal government should 
address regulatory, standards, and conformity assessment issues that have emerged or moved to the 
forefront since the Circular was last promulgated in 1998. OMB received approximately 70 
comments from a wide range of stakeholders, including companies, trade associations, academics, 
public interest groups, standards developing bodies, conformity assessment bodies, and private 
citizens. 1 

In response to those comments from 2012, OMB issued a Request for Comment on a Proposed 
Revision of OMB Circular A-l 19, “Federal Participation in the Development and Use of Voluntary 
Consensus Standards and in Conformity Assessment Activities,” on February 11, 2014 (79 Fed. 

Reg. 8207). OMB received over 80 comments from a wide range of stakeholders including 
companies and trade associations, academics, public interest groups, standards development bodies, 
conformity assessment bodies, and private citizens. 2 

Discussion and Responses to Significant Comments 

Although some commenters were critical of specific aspects of the proposed revision, the majority 
of commenters expressed support for the overall revisions to the Circular and the approaches taken. 
The more substantive comments are summarized below, along with OMB’s responses. 

While OMB carefully considered the proposals and concerns of all commenters, many comments 
were not germane, given the scope of the Circular. The purpose of the Circular is only to provide 
guidance on agency use of standards at a level that is practical and useable for all agencies. 

For example, OMB received particularly detailed or individualized comments regarding conformity 
assessment requirements (OMB defers to NIST and its anticipated revisions to its Guidance on 
Federal Conformity Assessment), country-specific standards policy (OMB defers to USTR), and the 
publishing requirements for guidance and regulations for agencies (OMB defers to the Office of the 
Federal Register). The most current version of NIST’s conformity assessment guidance is available 
at http ://www.nist.gov/standardsgov/. 

Based on OMB’s consideration and responses to these public comments, as well as developments in 
regulatory, standards, conformity assessment, and trade policy described above, the revised Circular 
A-l 19: 


1 Public comments submitted in response to OMB’s RFI can be found at 
http://www regulations.gov/#! documentDetail;D=OMB-2012-0003-0001. 

2 Public comments submitted in response to the Proposed Revisions can be found at 
http://www regulations.gov/#! documentDetail;D=OMB-2014-0001 -0001. 
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• provides additional guidance for agency participation in standards development activities, 
including with respect to serving on standards technical committees as well as the boards of 
standards developing bodies; 

• provides factors for agencies to consider when evaluating whether to use a standard to meet 
agency needs; 

• strengthens the role of agency Standards Executives to encourage better internal 
coordination and training on standards; 

• updates the provisions on how the U.S. Government manages and reports on the 
development and use of standards; 

• sets out factors for agencies to consider when assessing the effectiveness of conformity 
assessment options and determining the type(s) of conformity assessment program(s) to 
employ; 

• provides guidance to agencies on how they should implement provisions of the Circular in 
their rulemakings and guidance documents; 

• encourages each agency to alert the public through the Federal Register or the agency’s 
webpage when the agency is considering whether to participate in the standards 
development process of a particular body. Such notification would give the public notice 
that the agency may use the resulting standard in support of significant regulatory action, 
international regulatory cooperation activities or to otherwise address issues of national 
priority; 

• provides guidance to agencies on what factors to consider when incorporating standards by 
reference in regulation; 

• requires agencies to utilize the retrospective review mechanism set out in Executive Orders 
13563 and 13610, to ensure, among other things, that regulations incorporating standards by 
reference are updated on a timely basis; 

• encourages agencies to work together to reference the same version of standards in 
regulation and procurements and coordinate on conformity assessment requirements, where 
feasible; 

• incorporates references to trade-related statutory obligations on standards-related measures 
and directs Federal agencies to consult with USTR on how to comply with international 
obligations with regard to standards, technical regulations, and conformity assessment; and 

• maintains a strong preference for using voluntary consensus standards over government- 
unique standards in Federal regulation and procurement. 
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The Circular does not preclude the use of standards other than voluntary consensus standards in 
rulemaking, procurement or other program activities in cases where: voluntary consensus standards 
do not exist or where use of existing voluntary consensus standards would be inconsistent with law 
or otherwise impractical, including where use of a voluntary consensus standard would not be as 
effective at meeting the agency’s regulatory, procurement, or program needs. The Circular also 
recommends that the agency consider allowing the use of other standards as alternative means for 
complying with agency regulatory, procurement, or program requirements based on voluntary 
consensus standards where such other standards are also found to be suitable under the agency’s 
analysis. 

Promoting Voluntary Consensus Standards : Co mm ents received overwhelmingly supported a 
continued preference for Federal agencies’ use of voluntary consensus standards in lieu of 
developing their own standards, in line with the existing Circular and the requirements of Section 
12(d) of the NTTAA. 

There was also significant support among the commenters for providing Federal agencies the 
flexibility to choose among standards developed in the private sector, particularly standards 
developed for emerging technologies that may or may not precisely follow the traditional voluntary 
consensus standards development process. As such, and in order for agencies to choose the best 
standards for their mission needs, this policy maintains flexibility for agencies to use standards 
developed in the private sector that best meet agencies’ regulatory, procurement, or program needs, 
whether or not those standards are developed by voluntary consensus standards bodies. To assist 
agencies in deciding which standards to use, the Circular identifies factors to consider including, 
among others, the effectiveness and suitability of the standard for meeting agency regulatory, 
procurement, or program needs, the extent to which a standard meets the definition of a “voluntary 
consensus standard,” and whether the standard is reasonably available. 

Along these lines, many respondents provided valuable comments regarding the revised definitions 
in the Circular. The revised Circular incorporates a number of these suggestions intended to clarify 
the qualifications for a voluntary consensus standards development process. 

Several commenters asked for clarification on how the Circular addresses “leadership standards” (a 
term used to describe standards that exceed minimum threshold requirements). The Circular does 
not view leadership standards as different from other types of standards. As always, the Circular 
will indicate a preference for voluntary consensus standards over government-unique standards, and 
agencies may build on those standards as necessary to fulfill their obligations. 

In addition, the Circular does not indicate a preference for one type of conformity assessment 
procedure (e.g ., supplier’s declaration of conformity, third-party certification) over others. 

Agencies must have flexibility to determine the most appropriate conformity assessment program to 
meet agency needs, consistent with law (see Section 7 of the Circular). 

OMB further notes the Circular serves as a floor—not a ceiling—for when and how an agency 
should use standards. Agencies can, and in many cases will, have additional guidance for agency 
components or programs based on mission needs and other factors. 

Updating Considerations for Conformity Assessment : Commenters who addressed this issue 
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expressed moderate to very strong support for OMB providing guidance to agencies with respect to 
conformity assessment. Commenters recommended that the Circular encourage agencies to rely on 
private sector conformity assessment activities; provide criteria for agencies to utilize when 
determining how to best meet their needs; and help agencies comply with relevant international 
obligations regarding conformity assessment. Since the Circular was last published in 1998, the 
evolution of an increasingly globalized marketplace has induced growth in the scope and 
importance of conformity assessment. Global supply and distribution chains may mean a greater 
number of variables for agencies to assess and more complex logistics to account for when 
conducting assessments of conformity. Accordingly, the revised Circular encourages agencies to 
consider leveraging existing private sector conformity assessment activities wherever consistent 
with law and mission. 

Ensuring Compliance with International Obligations : Numerous commenters on the RFI and 
the Proposed Revision support OMB guidance to agencies on how to comply with relevant 
international obligations, including the WTO Agreement on Technical Barriers to Trade and other 
trade agreements. Accordingly, OMB in coordination with USTR, which has statutory authority in 
this area pursuant to 19 U.S.C. § § 2171 & 2541, has updated the Circular to reflect this need. 

The Circular directs Federal agencies to consult with USTR on how to comply with international 
trade obligations relating to standards, technical regulations and conformity assessment, including 
those codified at 19 U.S.C. § 2531. In addition, the revisions direct Federal agencies to consult with 
the State Department, to the extent international obligations other than trade obligations may be 
implicated. With respect to the Federal government’s formulation of international policies on 
standards and conformity assessment, regulation, and trade, the revisions would encourage, 
consistent with Executive Order 13609, greater coordination between the Interagency Committee on 
Standards Policy, the Regulatory Working Group, the Trade Policy Staff Committee and its 
subcommittees, and any other relevant interagency groups or committees. 

Several commenters to the Proposed Revision voiced concerns over specific issues regarding 
reciprocity of standards use or recognition by other countries or trade unions. Since these 
comments fall outside the Circular’s scope of providing guidance to agencies about the Federal 
government’s use of standards, OMB encourages parties to address such concerns directly with 
USTR. 

Agency Considerations for Incorporation By Reference : A wide variety of commenters 
representing public interest groups, individuals, citizens, standards development organizations 
(SDOs), and companies provided co mm ent on the question of incorporation by reference. Some 
comments advocated for the maintenance of current law and policy, noting many SDOs already 
freely publish their standards online in read-only formats, while others urged all standards 
incorporated by reference be made freely available on the Internet. 

The majority of these comments turned on particular interpretations of the “reasonably available” 
requirement in the Freedom of Information Act (FOIA), which requires that standards incorporated 
into regulation by reference be made reasonably available to the “class of persons affected.” As 
noted in the Draft Revision, it is not within the purview of the Circular to define reasonable 
availability. Rather, it is by statute the responsibility of the Office of the Federal Register (OFR) to 
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address this issue. 3 OFR revised its regulations on incorporation by reference in a final rule 
published November 7, 2014 at 79 Fed. Reg 66267. In that rule, OFR balanced its statutory 
obligations regarding reasonable availability of the standards with: (1) U.S. copyright law, (2) U.S. 
international trade obligations, and (3) agencies' ability to substantively regulate under their 
authorizing statutes. To address all of these obligations, the OFR rule required that agencies discuss 
how materials incorporated by reference were made available to parties (and where those materials 
are located) and to provide a summary of those materials in the preambles of their rulemaking 
documents. These requirements oblige agencies to provide more information on how they make 
material incorporated by reference available and a summary of the material, so readers can, if they 
like, find and review the standards. 

The revised Circular provides a non-exclusive set of factors to help agencies assess the availability 
of specific standards for particular regulatory contexts. These considerations may therefore inform 
the larger agency decision of whether to request approval for incorporation of those standards in 
regulation. As explained in the revised Circular, the basis of these factors for consideration is the 
Administrative Conference of the United States (ACUS) Recommendation 2011-5, Incorporation 
by Reference, 77 Fed. Reg. 2257 (January 17, 2012). 4 ACUS adopted the Recommendation through 
a consensus process in which a broad array of stakeholders participated, including standards 
developing bodies, academics, public interest groups, regulators, and industry. Based on the 
comments, the revised Circular bases its policy on the ACUS Recommendation, modified to 
enhance clarity. 

In general, the factors based on the ACUS Recommendation were well received, particularly the 
provision by SDOs of read-only access to their standards. OMB received numerous comments from 
SDOs highlighting the widespread existence of such a practice. In response to comments, OMB 
amended the Circular to recommend that agencies also consider the ease of access, registration, and 
navigation on those sites. OMB also received many comments on the option to provide a “freely 
available, non-technical summary.” Some commenters expressed the need for greater clarity as to 
what this might entail. Other commenters questioned the feasibility or utility of a non-tec hn ical 
summary for lengthy, complex, and/or highly technical standards. To clarify, OMB modified this 
factor to encourage agency discretion and collaboration with SDOs to enhance practical, impactful 
improvements to availability on a case-by-case basis. 

Additionally, OMB made clarifications emphasizing that: (1) the absence of one or more of such 
factors should not prevent an agency from using a particular standard; (2) agencies should work 
closely with SDOs to determine appropriate access to the standards for stakeholders; and (3) 
agencies should explain the reasoning for their choice of standard, including an explanation of how 
that standard may be accessed by stakeholders. 

Strengthening Guidance for Agency Participation in the Standards Development Process and 

the Role of Agency Standards Executives : Commenters expressed wide support for increased 
guidance for agency participation, including in standards development voting and standards body 
governance. As a floor, the revised Circular encourages agencies to participate fully in the 
standards development process—through voting as well as standards body governance—as equal 


3 See 5 U.S.C. §552(a)(l). 

4 See http://www.gpo.gov/fdsvs/pkg/FR-2012-01-17/html/2012-621 .htm. 
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parties. OMB, however, defers to individual agencies on their policies for determining to what 
extent and under what circumstances agency representatives are authorized to engage in 
particular activities, based on agency requirements and priorities. Additionally, the Circular 
encourages strong coordination by Agency Standards Executives (ASEs) to allow for effective 
use of agency resources in the standards development process. 

In response to requests for greater notice by agencies of their involvement in the standards 
development process, the revised Circular provides guidance to agencies on how they should 
discuss implementation of the Circular in their rulemakings and guidance documents. The 
revised Circular also encourages each agency to alert the public through the Federal Register 
and/or its webpage when the agency is considering whether to participate in the standards 
development process of a particular body, with the intention of using the resulting standard to 
support significant regulatory action, promote international regulatory cooperation activities, or 
address issues of national priority. Many of the comments regarding agency participation 
included agency-specific details that are beyond the scope of the Circular. OMB encourages 
commenters to contact agencies directly with feedback regarding their voting, travel, or any other 
policies. 

Ensuring the Timely Updating of Regulations and Other Agency Materials Incorporating 

Standards : Many commenters expressed views on whether the most current version of a standard 
should always be the one used by an agency. OMB recognizes that agencies may have good 
reasons for not using the most recent version of a standard. For example, if use of a revised 
standard significantly increases compliance costs that are not justified by the benefits of using the 
standard, the agency should be free to decline to adopt it. 

To ensure the timely updating of standards, the revised Circular requires agencies to utilize the 
retrospective review mechanism set out in Executive Orders 13563 and 13610. Many commenters 
voiced strong support for such periodic review, with many agreeing with the suggested three-to-five 
year review timeframe. OMB believes the use of retrospective review, as outlined by Executive 
Orders 13563 and 13610, can help ensure (1) the optimal version of a standard is being used by 
agencies; (2) agencies minimize the impact of conflicting requirements on their shared regulated 
entities due to references to different versions of standards; and (3), more broadly, the need for the 
standard and the regulation referencing it still exists. 

Additionally, in response to comments, OMB encourages agencies to work closely with SDOs to 
ensure agencies are aware of, and thus able to consider, updates and alternatives to existing 
standards. OMB also encourages agencies to work together to reference the same version of 
standards in regulation and procurements, consistent with statute and agency mission and 
objectives. 

OMB also received comments on specific provisions of the Proposed Circular. Below are the 
significant comments OMB received and our responses. 

Section-by-Section Discussion 

Proposed Section 3e—Definition of Voluntary Consensus Standard. Final Section 2d. 
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Several clarifications have been made to the definition to increase its technical accuracy (no 
substantive changes are intended). A commenter suggested that the definition, which states 
that a voluntary consensus standard needs to be either developed or adopted by a voluntary 
consensus standards body, be clarified to make clear that the standard at issue was developed 
or adopted through the use of a voluntary consensus standards development process. OMB 
agrees and has clarified the final Circular accordingly. 

Several other clarifications have been made. We have clarified that it is the voluntary 
consensus standard bodies, not the standards, which have intellectual property policies. We 
have also clarified that the Reasonable and Non-discriminatory (RAND) license obligations 
extend to “implementers of the standard” (this replaces the term “all interested parties,” 
which is a term of art in the rulemaking context). 

Lastly, the Federal government does not make a distinction between standards bodies based 
on where they are domiciled, but rather with respect to the attributes that characterize their 
processes for standards development. Thus, we have deleted the phrase “both domestic and 
international.” This edit is not intended to make a change in terms of coverage (the Circular 
includes a discussion of “international standards” at Section 5h and 5i). 

Proposed Section 3f(i)—Definition of Openness. Final Section 2e(i). 

Many commenters expressed concern with the potential burden of the “at all stages” part of 
the clause requiring SDOs to provide meaningful opportunities to participate. Several 
commenters noted that, as a practical matter, this need is already covered by the 
requirements for transparent procedures or processes and for due process. In response, 

OMB modified the final Circular to omit the reference to “at all stages” and clarify that 
opportunities to participate in standards development be on a non-discriminatory basis. 

Proposed Section 3f(ii)—Definition of Balance of Representation. Final Section 2e(ii)—Definition 

of Balance. 

Several commenters questioned the utility of changing this element from “balance of 
interest,” found in the current version of Circular A-l 19 as well as the American National 
Standards Institute’s (ANSI) Essential Requirements, to “balance of representation.” OMB 
intended for the definition to be consistent with the concepts of both those documents and 
has simplified the term to “balance” to avoid confusion and to allow for flexibility in how 
balance is determined during the consensus phase of the development or adoption of the 
standard. 

Additionally, some commenters expressed concern with the ambiguity of “sectors” in this 
definition. Others commented that a key objective of “balance” is preventing a single 
interest from dominating the decision-making process. OMB agrees and modified the final 
Circular, adapting suggested insertions to the text of the definition. 

Proposed Section 3f(iii)—Definition of Due Process. Final Section 2e(iii). 
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Several commenters expressed concern with the burden and practicality of requiring “full 
access” to the views and objections of other participants in standards development. 
Commenters noted the risks of micromanagement and unnecessary overreach. OMB’s 
intention is to ensure an adequate record of proceedings for the purpose of resolving 
objections and appeals, so we have modified the final Circular to reflect this intent. 

Proposed Section 6e—When deciding to use a standard, what are some of the factors my agency 
should consider? Final Section 5a. 

Several commenters expressed a desire for greater clarity in the relationship between agency 
missions and the preference for voluntary consensus standards over government-unique 
standards. OMB agrees, and therefore the Circular maintains the preference for voluntary 
consensus standards, per the NTTAA, while still maintaining flexibility for agencies to best 
meet their missions. 

Additionally, some commenters highlighted the utility of considering the extent to which a 
standard is already used by a Federal agency, by State and local governments, as well as the 
prevalence of the standard in the national and international marketplace. These are 
important considerations precisely aligned with Executive Orders 13563, 13609, and 13610, 
discussed above. Accordingly, OMB agrees, and modified the Circular to include these 
considerations. 

Proposed Section 4g—How should my agency determine whether a voluntary standard is 
‘‘reasonably available” in a regulatory or non-regulatory context? Final Section 5f — What factors 
should my agency consider to aid the determination of whether a standard is ‘‘reasonably 
available ” in a regulatory or non-regulatory context? 

In general, the factors based on the ACUS Recommendation were well received, particularly 
SDO provision of read-only access to standards. OMB received numerous comments from 
SDOs noting they already provide such access. One commenter, however, brought to 
OMB’s attention the disparate capabilities and requirements of various read-only standards 
websites. OMB agrees that this should be a consideration, and has amended the text to 
recommend agencies also consider the ease of access, registration, and navigation on those 
websites. 

As referenced above, OMB also received many co mm ents on the option to provide a “freely 
available, non-technical summary.” Some commenters expressed the need for greater clarity 
as to what this might entail. Other commenters questioned the feasibility or utility of 
providing a non-technical summary for lengthy, complex, and highly technical standards. 

To clarify, OMB modified this factor and potential option to encourage agency discretion 
and cooperation with SDOs to enhance practical, impactful improvements to availability on 
a case-by-case basis. 

Additionally, in response to several co mm ents asking for greater agency consideration of the 
accessibility of standards incorporated by reference, OMB revised the Circular to 
recommend that agencies provide, in the preamble of a Notice of Proposed Rulemaking 
(NPRM) or final rule or guidance document, an explanation of the incorporated materials; 
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how such incorporated standards may be accessed; and how the agency’s incorporation by 
reference of the standard would further the agency’s regulatory objective. 

Proposed Section 7—What is the Policy for Federal Participation in Voluntary Standards Bodies? 
Final Section 6 — What is the Policy for Federal participation in Voluntary Consensus Standards 
Bodies? 

Several commenters questioned the reformatting of this section from the existing Circular, 
specifically the absence of explicit authorization to provide “direct financial support” to 
standards development activities. Many agencies do so, consistent with applicable law, and 
here OMB defers to agency policy. To provide agencies with flexibility to meet their 
missions, the Circular maintains the existing language. 

Proposed Section 8a—What considerations should my agency make when it is determining the type 
of conformity assessment procedure(s) to use? Final Section 7a—What considerations should my 
agency make when it is addressing the need for conformity assessment? 

In response to comments, OMB simplified this section. OMB has incorporated the previous 
Section 8a, “Should my agency consider relying on private sector conformity assessment 
activities in conjunction with, or where appropriate, in lieu of governmental conformity 
assessment?” into one question which is now Section 7a, “What considerations should my 
agency make when it is addressing the need for conformity assessment?” 

Additionally several commenters expressed concern over the apparent distinction between 
using “international conformity assessment schemes” or “private sector conformity 
assessment activities” as was phrased in the proposed revision. OMB agrees that such 
international conformity assessment schemes are just one example of private sector 
conformity assessment activity, and modified the Circular accordingly. 

Accordingly, OMB Circular A-119 is revised as set forth below. 

Howard Shelanski 

Administrator, Office of Information and Regulatory Affairs 

EXECUTIVE OFFICE OF THE PRESIDENT 
Office of Management and Budget 

Washington D.C. 20503 

[insert date] 

Circular No. A-119 

Revised 
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CIRCULAR NO. A-119,Revised 


TO THE HEADS OF EXECUTIVE DEPARTMENTS AND AGENCIES 

SUBJECT: Federal Participation in the Development and Use of Voluntary Consensus Standards 
and in Conformity Assessment Activities 

TABLE OF CONTENTS 

BACKGROUND 

1. What is the Purpose of this Circular? 

DEFINITIONS 

2. What is a Standard? 

3. What is Conformity Assessment? 

POLICY 

4. To Whom Does This Policy Apply? 

5. What is the Policy for Federal Use of Standards? 

a. When deciding to use a standard, what are some of the factors my agency should consider? 

b. Does this policy establish a preference between voluntary consensus standards and other 
types of standards? 

c. When must my agency use voluntary consensus standards? 

d. What if no voluntary consensus standard exists? 

e. Should my agency give preference to performance standards? 

f. What factors should my agency consider to aid the determination of whether a standard is 
“reasonably available” in a regulatory or non-regulatory context? 

g. How should my agency reference standards? 

h. Are there standards-related international trade obligations that agencies must adhere to 
regarding the use of standards? 

i. When should my agency consider using or recognizing a standard used by a trading partner? 

j. How does this policy affect my agency's regulatory authorities and responsibilities? 

k. How does this policy affect my agency's procurement authority? 

l. What factors should my agency consider when determining whether to allow the use of more 
than one standard? 

m. How should my agency ensure that standards incorporated by reference in regulation are 
updated on a timely basis? 
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6. What is the Policy for Federal Participation in Standards Bodies? 

a. Must agency participants be authorized? 

b. Does agency participation indicate endorsement of any decisions reached by standards 
bodies? 

c. What forms of support may my agency provide? 

d. Do agency representatives participate equally with other members? 

e. How should my agency alert the public of its potential participation in standards 
development activities that could be used as a basis for rulemaking or other mission-related 
activities? 

f. What if a voluntary consensus standards body is likely to publish a suitable standard in time 
for an agency to use it? 

7. What is the Policy on Conformity Assessment? 

a. What considerations should my agency make when it is addressing the needs for conformity 
assessment? 

b. Are there statutory and international obligations concerning conformity assessment? 

c. What obligations does my agency have when considering whether to recognize a conformity 
assessment procedure in use in the market of a trading partner? 

d. How does this policy affect my agency’s regulatory authorities and responsibilities? 

e. When should my agency utilize the retrospective review mechanism with respect to 
conformity assessment? 

8. How will the U.S. Government Coordinate Regulatory Policy with Standards and Conformity 

Assessment Policy? 

MANAGEMENT AND REPORTING OF STANDARDS USE 

9. How Does My Agency Manage and Report on the Implementation of the Circular? 

10. What are the Procedures for Reporting My Agency’s Use of Standards in Regulations? 

11. What are the Procedures for Reporting My Agency's Use of Standards in Procurements? 

a. How does my agency report the use of standards in procurements on a categorical basis? 

b. How does my agency report the use of standards in procurements on a transaction basis? 

AGENCY RESPONSIBILITIES 

12. What are the Responsibilities of the Secretary of Commerce? 

13. What are the Responsibilities of the Heads of Agencies? 

14. What are the Qualifications and Responsibilities of Agency Standards Executives? 

SUPPLEMENTARY INFORMATION 

15. When will this Circular be reviewed? 

16. What is the Legal Effect of this Circular? 

17. Do You Have Further Questions? 
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BACKGROUND 


1. What is the Purpose of this Circular? 

This Circular establishes policies to improve the internal management of the Executive Branch 
with respect to the U.S. Gover nm ent’s role in the development and use of standards and 
conformity assessment. Consistent with section 12(d) of P.L. 104-113, the “National Technology 
Transfer and Advancement Act of 1995,” as amended (hereinafter “the NTTAA”), and U.S. 
Government executive orders, this Circular directs agencies to use standards developed or adopted 
by voluntary consensus standards bodies rather than government-unique standards, except where 
inconsistent with applicable law or otherwise impractical. The policies in this Circular are 
intended to facilitate agencies’ compliance with obligations under U.S. trade statutes and trade 
agreements. U.S. Federal law (19 U.S.C. § 2532) specifically prohibits any U.S. Government 
agency from engaging in standards-related activities that create unnecessary obstacles to the 
foreign commerce of the United States. 

This Circular provides guidance for agencies participating in the work of voluntary consensus 
standards bodies and describes procedures for satisfying the reporting requirements of the NTTAA. 
The policies in this Circular are intended to minimize the reliance by agencies on government- 
unique standards. The Circular also provides policy guidance to agencies on the use of conformity 
assessment in procurement, regulatory, and program activities. This Circular replaces Office of 
Management and Budget (OMB) Circular No. A-l 19, dated February 10, 1998. 

Many voluntary consensus standards are appropriate or adaptable for the Federal government's 
purposes. The use of such standards, whenever practicable and appropriate, is intended to achieve 
the following goals: 

(i) eliminating the cost to the Federal government of developing its own standards 
and decreasing the cost of goods procured and the burden of complying with 
agency regulation; 

(ii) providing incentives and opportunities to establish standards that serve national 
needs, encouraging long-term growth for U.S. enterprises and promoting 
efficiency, economic competition, and trade; and 

(iii) furthering the reliance upon private sector expertise to supply the Federal 
government with cost-efficient goods and services. 

This Circular will also help agencies meet the five fundamental strategic objectives for Federal 
engagement in standards activities set out in Memorandum M-12-08, “Principles for Federal 
Engagement in Standards Activities to Address National Priorities” 

(http://www.whitehouse.gov/sites/default/files/omb/memoranda/2012/m-12-08 l.pdf) , which was 
issued on January 17, 2012. 

While M-12-08 applies to Federal engagement in standards activities in those exceptional 
circumstances where the U.S. Government is playing a leading or convening role, OMB believes 
that the objectives are generally applicable to Federal engagement in standards and conformity 
assessment activities, absent contrary statutory requirements. 
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In addition, the Circular seeks to support efforts by the Federal government to ensure that: 

a. the benefits of regulations justify their costs, consistent with Executive Orders 
12866, “Regulatory Planning and Review,” and 13563, “Improving Regulation and 
Regulatory Review,” and OMB Circular A-4, “Regulatory Analysis”; 

b. agencies modify, streamline, expand, or repeal regulations that may be outmoded, 
ineffective, insufficient, or excessively burdensome through the retrospective review 
process, consistent with Executive Orders 13563 and 13610, “Identifying and Reducing 
Regulatory Burdens”; 

c. agencies work with their regulatory counterparts in other countries to address common 
health, safety, labor, security, or environmental issues, as well as unnecessary differences 
between the regulatory approaches of U.S. agencies and those of their foreign counterparts 
that may impair economic growth, innovation, competitiveness, and job creation, consistent 
with Executive Order 13609, “Promoting International Regulatory Cooperation”; and 

d. agencies consider the cumulative effects, including the cumulative burdens, of regulation, 
consistent with Executive Order 13610. 


DEFINITIONS 


2. What is a Standard? 


a. The term “standard,” or “technical standard,” (hereinafter “standard”) as cited in the 

NTTAA, includes all of the following: 

(i) common and repeated use of rules, conditions, guidelines or characteristics for 
products or related processes and production methods, and related management 
systems practices; 

(ii) the definition of terms; classification of components; delineation of procedures; 
specification of dimensions, materials, performance, designs, or operations; 
measurement of quality and quantity in describing materials, processes, products, 
systems, services, or practices; test methods and sampling procedures; formats for 
information and communication exchange; or descriptions of fit and measurements 
of size or strength; and 

(iii) terminology, symbols, packaging, marking or labeling requirements as they apply 
to a product, process, or production method. 

b. The term "standard" does not include the following: 

(i) professional standards of personal conduct; or 

(ii) institutional codes of ethics. 
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c. “Government-unique standard” is a standard developed by and for use by the Federal 
government in its regulations, procurements, or other program areas specifically for 
government use (i.e., it is not generally used by the private sector unless required by 
regulation, procurement, or program participation). The standard was not developed as a 
voluntary consensus standard as described in Sections 2d and 2e. 

d. “Voluntary consensus standard” is a type of standard developed or adopted by voluntary 
consensus standards bodies, through the use of a voluntary consensus standards 
development process as described in Section 2e. These bodies often have 
intellectual property rights (IPR) policies that include provisions requiring that 
owners of relevant patented technology incorporated into a standard make that 
intellectual property available to implementers of the standard on non- 
discriminatory and royalty-free or reasonable royalty terms (and to bind subsequent 
owners of standards essential patents to the same terms). In order to qualify as a 
“voluntary consensus standard” for the purposes of this Circular, a standard that includes 
patented technology needs to be governed by such policies, which should be easily 
accessible, set out clear rules governing the disclosure and licensing of the relevant 
intellectual property, and take into account the interests of all stakeholders, including the 
IPR holders and those seeking to implement the standard. 

e. “Voluntary consensus standards body” is a type of association, organization, or technical 
society that plans, develops, establishes, or coordinates voluntary consensus standards 
using a voluntary consensus standards development process that includes the following 
attributes or elements: 

(i) Openness: The procedures or processes used are open to interested parties. Such 
parties are provided meaningful opportunities to participate in standards 
development on a non-discriminatory basis. The procedures or processes for 
participating in standards development and for developing the standard are 
transparent. 

(ii) Balance: The standards development process should be balanced. Specifically, 
there should be meaningful involvement from a broad range of parties, with no 
single interest dominating the decision-making. 

(iii) Due process: Due process shall include documented and publically available 
policies and procedures, adequate notice of meetings and standards development, 
sufficient time to review drafts and prepare views and objections, access to views 
and objections of other participants, and a fair and impartial process for resolving 
conflicting views. 

(iv) Appeals process: An appeals process shall be available for the impartial handling 
of procedural appeals. 

(v) Consensus: Consensus is defined as general agreement, but not necessarily 
unanimity. During the development of consensus, comments and objections are 
considered using fair, impartial, open, and transparent processes. 

See Section 5h of this Circular for a discussion of “international standards.” 
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3. What is Conformity Assessment? 

Conformity assessment” is a demonstration, whether directly or indirectly, that specified 
requirements relating to a product, process, system, person, or body are fulfilled. Conformity 
assessment includes sampling and testing, inspection, supplier’s declaration of conformity, 
certification, and management system assessment and registration. Conformity assessment also 
includes accreditation of the competence of those activities. 


POLICY 


4. To Whom Does This Policy Apply? 

This Circular applies to all agencies and agency representatives who use standards or conformity 
assessment and/or participate in the development of standards. “Agency” means any executive 
department, independent commission, board, bureau, office, government-owned or controlled 
corporation, or other establishment of the Federal government. It also includes any regulatory 
commission or board, except for independent regulatory commissions insofar as they are subject to 
separate statutory requirements regarding the use of voluntary consensus standards. It does not 
include the Legislative or Judicial branches of the Federal government. 

5, What is the Policy for Federal Use of Standards? 

Consistent with Section 12 (d)(1) of the NTTAA, all Federal agencies must use voluntary 
consensus standards in lieu of government-unique standards in their procurement and regulatory 
activities, except where inconsistent with law or otherwise impractical. In these circumstances, 
your agency must submit a report describing the reason(s) for its use of government-unique 
standards in lieu of voluntary consensus standards as explained in Sections 9-11. 

a. When deciding to use a standard, what are some of the factors my agency should 
consider? 


(i) In evaluating whether to use a standard, which should be done on a case-by-case basis, 
an agency should consider the following factors, where applicable: 

(1) Whether the standard is effective and otherwise suitable for meeting agency 

regulatory, procurement, or program needs, including as applicable: 

(a) the nature of the agency’s statutory mandate and the consistency of the 
standard with that mandate; 

(b) the level of protection the standard provides or is expected to provide for public 
health, welfare, safety, and the environment; 

(c) the clarity and detail of the standard's language, as the wording of a standard 
may contain too much detail as well as too little; 
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(d) the costs and benefits of implementing other available standards that may also 
meet the agency’s needs; 

(e) the costs and benefits to the Federal government and the regulated public of the 
agency developing its own standard; 

(f) the ongoing use of the standard by other agencies for the same or a similar 
requirement, the use of which in a particular instance would increase 
consistency across the Federal government; 

(g) the ongoing use of the standard by State and local governments for the same or 
a similar requirement, the use of which in a particular instance would increase 
consistency across jurisdictions; 

(h) the prevalence of the use of the standard in the national and international 
marketplaces; 

(i) the problems addressed by the standard and changes in the state of knowledge 
and technology since the standard was prepared or last revised; 

(j) the extent to which the standard establishes performance rather than design 
criteria, where feasible; 

(k) the ability of small- and medium-sized enterprises (SMEs) to comply with the 
standard; and 

(l) the agency’s ability to use, and enforce compliance with, the standard in its 
regulatory process. 

(2) The extent to which, when preparing the standard, the standards body reflected the 
attributes of a voluntary consensus standards body set out in Section 2e of the 
Circular. The procedures related to these attributes ( e.g ., openness, balance, due 
process, appeals process, and consensus) should be easily accessible, clear and 
unambiguous. 

(3) The extent to which the standard is an international standard (see Section 5h). 

(4) Any barriers to membership and participation in the standards development 
process, given that fee structures, modes of participation, and other factors can 
impact the ability of SMEs, public interest groups, and the general public to 
participate. 

(5) Whether the standard is “reasonably available.” See Section 5f of the Circular for 
additional information. 

(6) The standards maintenance process used by the relevant standards developing body 
or bodies. 

(ii) If an agency is proposing to use a standard in a proposed or final rulemaking, the 

agency must comply with the "Principles of Regulation" (enumerated in Section 1(b) 
of Executive Order 12866, “Regulatory Planning and Review”), the analytical 
requirements of other relevant executive orders, and other documents as applicable, 
including those described in Section 1 of the Circular. 

(iii) As a general matter, standards being considered for use in regulation that specify 
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nomenclature, basic reference units, or methods of measurement or testing, and that 
are primarily empirical in their formulation, will ordinarily warrant less scrutiny by an 
agency than standards that embody factors that are less objective. 

(iv) An agency should, to the extent permitted by law, take account of the effect of using 
the standard on the economy, and of applicable Federal laws and policies, including 
laws and regulations relating to antitrust, national security, small business, product 
safety, environment, metrication, technology development, international trade, 
intellectual property and copyright, privacy and security, and conflicts of interest. 

(v) An agency should take into consideration the standards developer’s intellectual 
property rights (IPR) policies. Many standards developing bodies have policies 
requiring participating IPR holders to commit to license any relevant patented 
technology incorporated into a standard on non-discriminatory and royalty-free or 
reasonable royalty terms and to bind subsequent owners of standards-essential patents 
to the same terms. Such policies should be easily accessible, set out clear rules 
governing the disclosure and licensing of the relevant IPR, and take into account the 
interests of all stakeholders, including the IPR holders and those seeking to implement 
the standard. 

b. Does this policy establish a preference between voluntary consensus standards and 
other types of standards? Consistent with Section 12(d)(1) of the NTTAA, this policy 
establishes a preference for the use of voluntary consensus standards in lieu of government- 
unique standards. The Circular does not preclude the use of other standards in rulemaking, 
procurement, or other program activities in cases where voluntary consensus standards do not 
exist or use of existing voluntary consensus standards would be inconsistent with law or 
otherwise impractical, including where use of a voluntary consensus standard would not be as 
effective at meeting the agency’s regulatory, procurement or program needs. The Circular 
also recommends that the agency consider allowing the use of other standards as alternative 
means for complying with agency regulatory, procurement, or program requirements that 
incorporate voluntary consensus standards, where such other standards are also found to be 
suitable under the agency’s analysis. See Section 51 concerning the selection of multiple 
standards. 

c. When must my agency use voluntary consensus standards? Your agency must use 
voluntary consensus standards in its regulatory, procurement, and program activities in lieu of 
government-unique standards, unless use of such standards would be inconsistent with 
applicable law or otherwise impractical. In all cases, your agency has the discretion to 
decline to use existing voluntary consensus standards if your agency determines that such 
standards are inconsistent with applicable law or otherwise impractical, and instead to use a 
government-unique standard or other standard. 

In addition to consideration of voluntary consensus standards, this Circular recognizes the 
contributions of standardization activities that take place outside of the voluntary consensus 
standards process. Therefore, in instances where use of voluntary consensus standards 
would be inconsistent with applicable law or otherwise impracticable ( e.g ., no voluntary 
consensus standard would be effective in meeting agency regulatory, procurement, or 
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program needs), agencies should consider, to the extent consistent with applicable law - as 
an alternative to using a government-unique standard - other standards that meet the 
agency’s regulatory, procurement or program needs, deliver favorable technical and 
economic outcomes (such as improved interoperability) and are widely utilized in the 
marketplace. 

In those circumstances where an agency elects to use or develop a government-unique 
standard in lieu of using a voluntary consensus standard, Section 12(d) of the NTTAA 
requires the agency to submit a report describing the reason(s) to OMB. Under the Circular, 
this report is submitted to OMB through the National Institute of Standards and Technology 
(NIST). For more information on reporting, see Sections 9-11 of this Circular. 

(i) “Use” means incorporation of a standard in whole, in part, or by reference for 
procurement purposes; inclusion of a standard in whole, in part, or by reference in 
regulation(s); or inclusion of a standard in whole, in part, or by reference in other 
mission-related activities. 

(ii) “Impractical” includes circumstances in which such use would fail to serve the 
agency's regulatory, procurement, or program needs; be infeasible; be inadequate, 
ineffectual, inefficient, or inconsistent with the agency mission or the goals of using 
voluntary consensus standards; be inconsistent with a provision of law; or impose more 
burdens, or be less useful, than the use of another standard. 

d. What if no voluntary consensus standard exists? In cases where no suitable voluntary 
consensus standards exist, an agency may use suitable standards that are not developed by 
voluntary consensus bodies. In addition, an agency may develop its own standards or use 
other government-unique standards, solicit interest from qualified standards development 
organizations for development of a standard, or develop a standard utilizing the process 
principles outlined in Section 2e of the Circular. As explained above (see Section 5c of the 
Circular), Section 12(d) of the NTTAA provides that an agency may use a government- 
unique standard in lieu of a voluntary consensus standard if the use of a voluntary consensus 
standard would be inconsistent with applicable law or otherwise impractical. In such cases, 
the agency must file the statutorily required report (see Section 10). 

e. Should my agency give preference to performance standards? Yes. Pursuant to Section 
1(b)(8) of Executive Order 12866 and 19 U.S.C.§ 2532(3), your agency should give 
preference to performance standards where feasible and appropriate. The term “performance 
standard” refers to a standard that states requirements in term s of required results, but without 
stating the methods for achieving the required results. A performance standard may define 
the functional requirements for an item, operational requirements, and/or interface and 
interchangeability characteristics. A prescriptive standard, by contrast, may specify design 
requirements, such as materials to be used, how a requirement is to be achieved, or how an 
item is to be fabricated or constructed. It is important to recognize that, in many instances, a 
standard may contain both performance and prescriptive elements. In such cases, agencies 
should select standards that provide the most flexibility for achieving the desired results. In 
most instances, these will be standards that rely more heavily on performance-based criteria. 

f. What factors should my agency consider to aid the determination of whether a standard 
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is “reasonably available” in a regulatory or non-regulatory context? In determining 
whether a standard is “reasonably available” to regulated and other interested parties, 
agencies should take into account the following factors, given that reasonable availability is 
context-specific. The absence of one or more of these factors alone should not be used as a 
basis for an agency decision not to use the standard. This section shall also be applied in a 
manner consistent with U.S. international obligations to use relevant international standards 
(see Section 5h of the Circular); the “Principles of Regulation” (enumerated in Section 1(b) 
of Executive Order 12866); and the need to “protect public health, welfare, safety, and our 
environment while promoting economic growth, innovation, competitiveness, and job 
creation” (see Section 1 of Executive Order 13563). 

Factors to consider include: 

(i) whether the standards developer is willing to make read-only access to the standard 
available for free on its website during the comment period to facilitate more effective 
access, because access may be necessary during rulemaking to make public 
participation in the rulemaking process effective; 

(ii) the cost to regulated and other interested parties to access a copy of the material, 
including the cumulative cost to obtain incorporated materials, and their ability to bear 
the costs of accessing such materials in a particular context; 

(iii) the extent particular access is needed to achieve agency policy or to subject the 
effectiveness of agency programs to public scrutiny; and 

(iv) whether the standards developer can provide a summary that explains the content of 
the standard in a way that meets agency needs and is understandable to a member of 
the public who lacks relevant technical expertise. 

When considering incorporation by reference, agencies should include in the preamble of an 
NPRM, final rule, or guidance document an explanation of the incorporated materials, and 
how the agency’s incorporation by reference of the standard would further the agency’s 
regulatory objective (see 79 FR 66267 published November 7, 2014 entitled Incorporation 
by Reference). 

If an agency incorporates by reference material that is already freely available, the agency 
should ensure that the material is available electronically in a location where regulated and 
other interested parties will be able to find it easily by, for example, providing a link to the 
website of the standards body. If an agency incorporates by reference material that is 
copyrighted or otherwise subject to legal protection and not freely available, the agency 
should work with the relevant standards developer to promote the availability of the 
materials, consistent with applicable law, such as through the use of technological solutions, 
low-cost-publication, or other appropriate means, while respecting the copyright owner’s 
interest in protecting its intellectual property. To this end, the agency should explain, in the 
preamble of an NPRM, final rule, or guidance document how such incorporated standards 
may be accessed consistent with OFR’s regulations in 1 CFR Part 51. 


g. How should my agency reference standards? Where your agency seeks to incorporate a 
standard by reference, your agency should reference the standard, along with sources from 
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which a copy of the standard may be obtained, in relevant publications, regulations, and 
related internal documents. The Office of the Federal Register’s regulations at 1 CFR Part 
51 govern the use of incorporation by reference in regulation. For all other uses, your 
agency must determine the most appropriate form of reference. If a standard is used and 
published in an agency document, your agency must observe and protect the rights of the 
copyright holder and meet any other similar obligations, such as those relating to patented 
technology that must be used to comply with the standard. 

h. Are there standards-related international trade obligations that agencies must adhere to 
regarding the use of standards? Yes. There are statutory and international obligations 
governing “standards-related activities.” In particular, agencies should be aware of the 
obligations in 19 U.S.C. § 2532, which provides that “[n]o Federal agency may engage in any 
standards-related activity that creates unnecessary obstacles to the foreign commerce of the 
United States...” and that “[e]ach Federal agency shall ensure, in applying standards-related 
activities with respect to any imported product, that such product is treated no less favorably 
than are like domestic or imported products...” “Standards-related activity” is defined in 19 
U.S.C. § 2571 and covers certain standards and technical regulations as well as conformity 
assessment procedures. These statutory requirements reflect U.S. international obligations 
under the World Trade Organization (WTO) Agreement on Technical Barriers to Trade (TBT 
Agreement). Specifically, Articles 2.1 and 2.2 and Ann ex 3 of the Agreement commit the 
United States, with respect to standards and technical regulations, to provide no less 
favorable treatment to imported products than like domestic products and to avoid 
unnecessary obstacles to trade. 

In addition, the United States is obligated under the TBT Agreement to use relevant 
international standards, except where such standards would be an ineffective or 
inappropriate means to fulfill the legitimate objective pursued. In particular, the TBT 
Agreement, Article 2.4, provides that: “Where technical regulations are required and 
relevant international standards exist or their completion is imminent, [WTO] Members shall 
use them, or the relevant parts of them, as a basis for their technical regulations except when 
such international standards or relevant parts would be an ineffective or inappropriate means 
for the fulfillment of the legitimate objectives pursued, for instance because of fundamental 
climatic or geographical factors or fundamental technological problems.” In addition, 19 
U.S.C. § 2532 directs Federal agencies, in developing standards, to base the standards on 
international standards if appropriate. 

The WTO TBT Agreement defines “technical regulation” as a document which sets out 
product characteristics or their related processes and production methods, including 
applicable administrative provisions, with which compliance is mandatory, and states that 
the definition may include or deal exclusively with terminology, symbols, packaging, 
marking or labeling requirements as they apply to a product, process, or production method. 
The WTO TBT Agreement excludes services, sanitary and phytosanitary measures, and 
government procurement from its scope. These areas are covered by other WTO agreements 
discussed later in this section. 

The WTO TBT Agreement does not provide a list of bodies that develop international 
standards. However, in 2000, the WTO Committee on Technical Barriers to Trade adopted 
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a decision setting out principles that standards bodies should follow when developing 
international standards (the Decision on Principles for the Development of International 
Standards, Guides and Recommendations with Relation to Articles 2, 5 and Annex 3 of the 
WTO Agreement on Technical Barriers to Trade, hereinafter the Committee Decision). 
Please see the annex to this Circular for additional information. 

In addition, certain U.S. free trade agreements commit the United States to determine 
whether an international standard exists based on the principles set out in the Committee 
Decision. Agencies should, therefore, pay close attention to the Committee Decision and 
consult with USTR when questions arise as to evaluating whether a standard developed by a 
particular standards body is “international.” A voluntary consensus standard may be an 
international standard under the WTO TBT Agreement or provisions of U.S. trade 
agreements applying to technical barriers to trade, if it was developed in accordance with the 
Committee Decision principles. 

The WTO Agreement on the Application of Sanitary and Phytosanitary Measures (SPS 
Agreement), which applies to sanitary and phytosanitary measures (e.g., food safety 
regulations), also addresses the use of relevant international standards. Article 3.1 of the 
WTO SPS Agreement states: “To harmonize sanitary and phytosanitary measures on as wide 
a basis as possible, [WTO] Members shall base their sanitary or phytosanitary measures on 
international standards, guidelines or recommendations, where they exist, except as 
otherwise provided for in this Agreement, and in particular paragraph 3.” Article 3.3 of the 
SPS Agreement also provides that: 

Members may introduce or maintain sanitary or phytosanitary measures which result 
in a higher level of sanitary or phytosanitary protection than would be achieved by 
measures based on the relevant international standards, guidelines or 
recommendations, if there is scientific justification, or as a consequence of the level 
of sanitary or phytosanitary protection a Member determines to be appropriate in 
accordance with the relevant provisions of paragraph 1 through 8 of Article 5. 
Notwithstanding the above, all measures which result in a level of sanitary or 
phytosanitary protection different from that which would be achieved by measures 
based on international standards, guidelines or reco mm endations shall not be 
inconsistent with any other provisions of this Agreement. 

The WTO SPS Agreement includes a detailed definition of “sanitary or phytosanitary 
measure” and generally covers measures related to food safety and the spread of disease or 
pests. Unlike the WTO TBT Agreement, the WTO SPS Agreement identifies three bodies, 
among others, that develop international standards, guidelines and recommendations for 
purposes of implementing the WTO SPS Agreement. In particular, the SPS Agreement 
identifies the following as international standards, guidelines and recommendations: for food 
safety, those established by the Codex Alimentarius Commission; for animal health and 
zoonoses, those developed under the auspices of the International Office of Epizootics; for 
plant health, those developed in the framework of the International Plant Protection 
Convention; and “for matters not covered by the above organizations, appropriate standards, 
guidelines and recommendations promulgated by other relevant international organizations 
open for membership to all [WTO] Members, as identified by the [SPS] Committee.” 
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The United States also has procurement obligations under the WTO Agreement on 
Government Procurement (GPA) and a number of Free Trade Agreements (see FAR Subpart 
25.4) to base the technical specifications on international standards, where available. Article 
X.2 of the WTO GPA provides: 

In prescribing the technical specifications for the goods or services being procured, a 

procuring entity shall, where appropriate: 

(i) set out the technical specification in terms of performance and functional 
requirements, rather than design or descriptive characteristics; and 

(ii) base the technical specification on international standards, where such exist; 
otherwise, on national technical regulations, recognized national standards or 
building codes. 

Agencies should consult with USTR on questions regarding international trade obligations 
relating to regulations, standards, conformity assessment procedures, or procurement or with 
respect to any requests from countries regarding the establishment of mutual arrangements 
with respect to standards-related activities. Pursuant to 19 U.S.C. § 2171 and 2541, USTR 
has statutory authorities and responsibilities in these areas including “for coordinating 
United States discussions and negotiations with foreign countries for the purpose of 
establishing mutual arrangements with respect to standards-related activities.” Agencies 
should consult with the Department of State on questions regarding the application of 
international agreements other than trade agreements. 

i. When should my agency consider using or recognizing a standard used by a trading 
partner? To the extent a standard used by a trading partner is an international standard or 
voluntary consensus standard, agencies should consider and use them as described in this 
Circular (see, e.g., Sections 5a and h). In addition, agencies should consider and use a 
standard used by a trading partner, where, pursuant to an international agreement, the United 
States has agreed to do so (e.g., pursuant to a mutual recognition agreement). Agencies 
should also examine how recognition or use of a standard used by a trading partner would 
impact the policy objectives set out in Section 1 of Executive Order 13609, “Promoting 
International Regulatory Cooperation.” An agency may have additional obligations to 
consider a standard used by a trading partner pursuant to Section 3(d) of Executive Order 
13609. 

In addition, the United States is obligated to adhere to certain international agreements with 
respect to accepting as equivalent standards used in the regulations of U.S. trading partners. 
In particular, the WTO TBT Agreement, Article 2.7, requires the United States and other 
WTO Members to “give positive consideration to accepting as equivalent technical 
regulations of other Members, even if these regulations differ from their own, provided they 
are satisfied that these regulations adequately fulfill the objectives of their own regulations.” 

The WTO SPS Agreement also contains obligations regarding equivalence determinations 
with respect to foreign country sanitary and phytosanitary measures (SPS measures). Article 
4 of the SPS Agreement provides: 
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(i) Members shall accept the sanitary or phytosanitary measures of other Members as 
equivalent, even if these measures differ from their own or from those used by 
other Members trading in the same product, if the exporting Member objectively 
demonstrates to the importing Member that its measures achieve the importing 
Member's appropriate level of sanitary or phytosanitary protection. For this 
purpose, reasonable access shall be given, upon request, to the importing Member 
for inspection, testing and other relevant procedures. 

(ii) Members shall, upon request, enter into consultations with the aim of achieving 
bilateral and multilateral agreements on recognition of the equivalence of 
specified sanitary or phytosanitary measures. Agencies should consult with USTR 
regarding entering into any such consultations. 

U.S. Federal law, as codified at 19 U.S.C. § 2578a, provides that an agency may not 
determine that a foreign country’s sanitary or phytosanitary measure is equivalent to a 
sanitary or phytosanitary measure established under the authority of Federal law unless the 
agency determines the foreign country measure provides the same level of sanitary or 
phytosanitary protection as the U.S. measure. 

See SPS Agreement, Ann ex A, for the definition of SPS measure. 

j. How does this policy affect my agency's regulatory authorities and responsibilities? 

This policy does not preempt or restrict agencies' authorities and responsibilities to make 
regulatory decisions authorized by statute. Such regulatory authorities and responsibilities 
include determining the level of acceptable risk and risk-management, and due care; setting 
the level of protection; and balancing risk, cost, and availability of alternative approaches in 
establishing regulatory requirements. However, agencies should consider using voluntary 
consensus standards, as described in this Circular, to achieve their regulatory, procurement, 
and program needs, including for test methods, sampling procedures, and protocols, if 
applicable. 

k. How does this policy affect my agency's procurement authority? This policy does not 
preempt or restrict agencies’ authorities and responsibilities to identify the capabilities that 
they need to obtain through procurements. Rather, this policy establishes a preference for 
using a voluntary consensus standard, to the extent a suitable one exists, instead of pursuing 
an identified capability through reliance on a government-unique standard. 

l. What factors should my agency consider when determining whether to allow the use of 
more than one standard? In considering whether to allow the use of more than one 
standard for suppliers to demonstrate that they meet a particular program, procurement, or 
regulatory requirement, your agency should consider whether doing so meets your agency’s 
regulatory, procurement, or program needs (see, e.g.. Section 5a). It may be appropriate for 
the agency to allow the use of multiple standards in order, for example, to permit greater 
flexibility for producers and service providers in meeting program, procurement, or 
regulatory requirements, enhance competition in the marketplace, provide greater choice to 
consumers, and enable new innovative solutions to be developed. Allowing the use of more 
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than one standard may also allow producers and service providers to meet both U.S. 
requirements and requirements in different markets simultaneously, thereby reducing burdens 
for manufacturers and service providers while effectively addressing agency objectives. 

m. How should my agency ensure that references to standards incorporated in regulation 
are updated on a timely basis? Consistent with Executive Orders 13563, “Improving 
Regulation and Regulatory Review,” and 13610, “Identifying and Reducing Regulatory 
Burdens,” agencies should, on a regular basis, review and if necessary update references to 
standards that have been incorporated by reference. Each agency should undertake a 
standards-specific review of such incorporated standards every three-to-five years, or when 
stakeholders otherwise provide adequate information that a standards-specific review is 
necessary due to urgent matters of health and safety, the need to remain current with 
technological changes, or for other compelling reasons. For example, regulated entities may 
petition agencies to update incorporated standards. 

In accordance with the retrospective review provisions of Executive Orders 13563 and 
13610, an agency should also consult with stakeholders prior to issuance of an NPRM, for 
example through an Advanced Notice of Proposed Rulemaking (ANPRM) or an RFI. Such 
consultations will help the agency identify which updated or new standards would 
potentially be controversial (if incorporated by reference in regulation) and which ones 
would not, and also whether incorporation of new or updated standards may warrant 
substantive changes to the underlying regulation. 

(i) In the case of standards for which updating or substituting a new standard or standards 
would not be controversial, the agency should, to the extent permitted by law, consider 
issuing a standards-specific direct final rule. 

(ii) In the case of other standards for which an agency may wish to update or substitute a 
new standard or standards, the agency should publish a standards-specific NPRM. 

(iii) To promote efficiency, and to the extent feasible and appropriate, an agency is 
encouraged to consolidate proposals to update or substitute standards, rather than 
conducting separate rulemakings for each standard the agency wants to update or 
substitute. 

(iv) In the case of standards for which updating or substituting a new standard would 
require a substantial re-opening of a rule, such revisions should be addressed in the 
context of a broader-scope “look-back” NPRM (rather than through a standards- 
specific NPRM), where such re-opening meets the objectives and criteria set out in 
Executive Orders 13563 and 13610. 

(v) If an agency decides not to adopt the most recent version of a standard, the agency 
should explain its reasons in the final rule. 

(vi) Except in urgent circumstances, and where consistent with law and international 
obligations, agencies must allow a reasonable interval between the publication of final 
rules and their effective dates, in order to provide affected stakeholders sufficient time 
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to adapt their products and methods of production to comply with the updated or new 
standard. 

6. What is the Policy for Federal Participation in Standards Bodies? 

Consistent with Section 12(d)(2) of the NTTAA, agencies must consult with voluntary 
consensus standards bodies and must participate with such bodies in the development of 
standards when consultation and participation is in the public interest and is compatible with 
their missions, authorities, priorities, and budgetary resources. In voluntary consensus standards 
development processes, agency participation can be an important contribution to ensuring 
balance is achieved. 

The WTO TBT and SPS Agreements also contain obligations regarding participation in 
international standards developing bodies: 

(i) TBT Agreement, Article 2.6: “With a view to harmonizing technical regulations on as 
wide a basis as possible, Members shall play a full part, within the limits of their 
resources, in the preparation by appropriate international standardizing bodies of 
international standards for products for which they either have adopted, or expect to 
adopt, technical regulations.” 

(ii) TBT Agreement, Article 5.5: “With a view to harmonizing conformity assessment 
procedures on as wide a basis as possible, Members shall play a full part, within the 
limits of their resources, in the preparation by appropriate international standardizing 
bodies of guides and recommendations for conformity assessment procedures.” 

(iii) SPS Agreement, Article 3.4: “Members shall play a full part, within the limits of their 
resources, in the relevant international organizations and their subsidiary bodies, in 
particular the Codex Alimentarius Commission, the International Office of Epizootics, 
and the international and regional organizations operating within the framework of the 
International Plant Protection Convention, to promote within these organizations the 
development and periodic review of standards, guidelines and recommendations with 
respect to all aspects of sanitary and phytosanitary measures.” 

Each agency should arrange for qualified representatives to participate in standards development 
activities, as appropriate. The representatives’ function should be to participate in the standards 
writing and/or, as appropriate, act as a liaison, provide information and expertise, and 
communicate the views of the agency concerning the standards being developed and the 
procedures being followed. 

a. Must agency participants be authorized? Agency representatives who, at Gover nm ent 
expense, participate in activities of standards bodies on behalf of the agency must do so as 
specifically authorized agency representatives. Agency support for, and participation by, 
agency representatives in standards bodies must be in compliance with applicable laws and 
regulations. For example, agency support is subject to legal and budgetary authority and 
availability of funds. Similarly, participation by agency representatives (whether or not on 
behalf of the agency) in the activities of standards bodies is subject to the laws and regulations 
that apply to participation by Federal employees in the activities of outside organizations. 
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Additionally, in considering the applicability of 18 U.S.C. § 208, which addresses Federal 
participation in outside organizations, the Office of Legal Counsel (OLC) of the Department of 
Justice has determined the following: “When the board of an outside organization plays an 
integral role in the process of setting standards, it would therefore frustrate the statute [NTTAA] 
to forbid Federal employees from being on the board. They could not then take the “active” role 
that Congress mandated. To carry out the statute, therefore, employees may serve on these 
outside boards without running afoul of 18 U.S.C. § 208, if the boards are engaged in the 
standard-setting activities in which Congress directed Federal agencies to participate.” 5 

Furthermore, in the 2001 amendment to the NTTAA (in Section 1115 of Public Law 107-107 
(which enacted a new paragraph of Section 12(d)), Congress expressly exempted application of 
5 U.S.C. § 5946 (which prohibits payments for membership and conference fees) with respect to 
activities of Federal agencies and personnel in carrying out Section 12(d) of the NTTAA (see 15 
U.S.C. § 272 note). As noted in the previous Section, active agency technical involvement and 
leadership in standards activities is encouraged by this Circular and the NTTAA. However, 
agency representatives should avoid the practice or the appearance of undue influence relating to 
their participation in standards bodies and activities. 

b. Does agency participation indicate endorsement of any decisions reached by standards 
bodies? Agency participation in standards bodies does not connote agency endorsement or 
agreement with decisions by such bodies. 


c. What forms of support may my agency provide to standards development? The forms of 
agency support may include: 

(i) direct financial support ( e.g ., grants, memberships, and contracts); 

(ii) administrative support (e.g., travel costs, hosting of meetings, and secretarial 
functions); 

(iii) technical support (e.g., cooperative testing for standards evaluation and participation of 
agency personnel in the activities of standards bodies); 

(iv) joint planning with standards bodies to promote identification and development of 
needed standards; and 

(v) participation of agency personnel. 

Federal agencies should give adequate priority in their budgets to the need for agency 

officials to participate in standards development. 

d. Do agency representatives participate equally with other members? Consistent with agency 
regulation and policy, agency representatives should participate actively and on an equal basis 
with other members, consistent with the procedures of standards bodies, particularly in matters 
such as establishing priorities, developing procedures for preparing, reviewing and approving 
standards, and developing or adopting new standards. Active participation includes full 
involvement in discussions and technical debates, registering of opinions and, if selected, 


5 See OLC’s opinion of August 24, 1998, on “Application of § 208 to Service by Executive Branch Employees on 
Boards of Standard-Setting Organizations,” http://oge.gov/DisplayTemplates/ModelSub.aspx7km826 . 
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serving as chairpersons (or other leadership positions) or in other official capacities. Agency 
representatives may vote, in accordance with the procedures of the standards body, at each stage 
of the standards development process, unless prohibited from doing so by law or their agencies. 

e. How should my agency alert the public of its potential participation in standards 
development activities that could be used as a basis for rulemaking or other mission- 
related activities? In the interest of enhancing transparency and information-sharing, agencies 
should advise the public about ongoing or planned participation in standards development 
activities, when, for instance, doing so to address issues of national priority, or in support of 
significant regulatory action or international regulatory cooperation activities. Methods could 
include publication of a notice in the Federal Register, providing notice on the agency’s public 
website, or using other appropriate mechanisms. The information provided could include, for 
example: 

(i) which body or organization is developing the standard; 

(ii) why the agency’s participation is relevant to the public; and 

(iii) methods by which the public can obtain more information. 

When an agency is unsure of the nature and extent of standards activity that may be relevant for 
an upcoming regulation, procurement, or non-regulatory action, the agency is encouraged to 
request information on standards development through, for example, an RFI or an ANPRM. 
Agencies could also use more informal means, such as contacting relevant bodies directly. 

Note that agency participation in the standards development process is not a prerequisite for 
incorporating or otherwise using a standard. 

f. What if a voluntary consensus standards body is likely to publish a suitable standard in 
time for an agency to use it? If a voluntary consensus standards body is in the process of 
developing or adopting a voluntary consensus standard that would likely be practical for an 
agency to use, and would likely be developed or adopted on a timely basis, an agency should not 
develop its own standard and instead should participate in the activities of the voluntary 
consensus standards body (or at least monitor the development of the standard) so that the 
agency is in a position to use the standard at the appropriate time. 

7. What is the Policy on Conformity Assessment? 

Section 12(b) of the NTTAA requires the NIST to coordinate Federal, State, and local standards 
activities and conformity assessment activities with private sector standards development and 
conformity assessment activities, with the goal of eliminating unnecessary duplication and 
complexity in the development and promulgation of conformity assessment requirements and 
measures. The Secretary of Commerce, through NIST, issues guidance to the agencies on 
conformity assessment (see www.standards.gov) . 

Building upon the NTTAA, U.S. statutory and international obligations, and NIST’s previous 
guidance to agencies on conformity assessment, agencies may develop and use conformity 
assessments procedures that: 
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(i) are effective and appropriate in carrying out agency missions; 

(ii) minimize the regulatory and/or conformity assessment burden on regulated entities; 

(iii) are in accordance with statutory and international obligations; 

(iv) conserve and leverage agency resources; and 

(v) increase acceptance of U.S. products in domestic and foreign markets. 

Agencies should be aware that conformity assessment procedures and systems can be used in 
many different ways to support their missions. Conformity assessment procedures and systems 
may be integral to a regulatory agency’s compliance and enforcement system, or the results of 
conformity assessment may be an input to the regulatory system. The agency may have roles 
and responsibilities related to the development and maintenance of conformity assessment 
system requirements and/or the conduct of specific activities in these systems. Flexibility is 
essential when devising an optimal conformity assessment system tailored to the missions of 
regulatory agencies. 

Agencies should also design conformity assessment programs with the objectives of furthering 
outcomes that are closely aligned with market dynamics and otherwise maximize net benefits to 
society. In this context, agencies should recognize the possible contribution of private sector 
conformity assessment activities. When properly conducted, conformity assessments conducted 
by private sector conformity assessment bodies can increase productivity and efficiency in 
government and industry, expand opportunities for international trade, conserve resources, improve 
health and safety, and protect the environment. 

Working closely with NIST and OMB, agencies are encouraged to identify their conformity 
assessment needs in such areas as regulatory compliance and enforcement, procurement, and 
other programmatic contexts and to assess whether the use of private sector conformity 
assessment mechanisms in lieu of or in conjunction with government conformity assessment 
procedures would be beneficial, where such use is feasible and appropriate and not otherwise 
prohibited by law. Agencies should also consider whether reliance on international conformity 
assessment schemes would meet their conformity assessment needs. 

Agencies will continue to have the flexibility of choosing conformity assessment programs, 
whether private, public, or some combination thereof, to best achieve the objectives above. 

a. What considerations should my agency make when it is addressing the need for 
conformity assessment? Agencies should consider appropriate agency roles and 
responsibilities to carry out their missions in a way that protects health, safety, welfare, and the 
environment, while promoting economic growth, competitiveness, and job creation, and 
reducing costs and burdens, including the cumulative effects of regulation, on the affected 
public and the U.S. economy. When an agency is evaluating whether to put in place a 
conformity assessment program, it should, consistent with statute, resource constraints, and the 
instruments set out in Section 1 of this Circular, consider certain factors when assessing the 
effectiveness of conformity assessment options and determining the type(s) of conformity 
assessment to employ. These include: 

(i) the objective(s) of the underlying regulation, procurement, or program activity; 

(ii) the level of confidence required by the agency to ensure that the agency objective(s) 

30 


JA2037 


Case l:13-cv-01215-TSC Document 169-1 Filed 02/08/16 Page 32 of 44 
USCA Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 279 of 573 


has/have been achieved, weighing the risk of non-compliance and its associated 
consequences with the anticipated costs of demonstrating compliance (including time 
and resources) to the producers, suppliers, consumers, and the agency; 

(iii) whether there are existing private sector conformity assessment activities, acceptance 
schemes or arrangements, that may work in conjunction with or, where appropriate, in 
lieu of governmental conformity assessment activities, except where such activities 
are inconsistent with law, unfit for regulatory or other agency purpose, or otherwise 
impractical; 

(iv) their consistency with statutory and international obligations; 

(v) consideration of the available scientific and technical information, as relevant to the 
selection of the appropriate conformity assessment program; 

(vi) relevant industry practice and experience, and the industry’s history of compliance; 

(vii) the need to reduce duplication and complexity, and ensure consistency and 
coordination with the conformity assessment approaches of other agencies, where 
feasible, appropriate, and consistent with law; 6 

(viii) the appropriateness of recognizing the results of private sector conformity assessment 
programs being utilized in State, local, and/or foreign government regulation, 
consistent with subsection (iii); and 

(ix) the degree of transparency to stakeholders and the public of the conformity 
assessment activity. 

b. Are there statutory and international obligations concerning conformity assessment? 

There are statutory and international obligations governing standards-related activities, which 
include conformity assessment procedures. In particular, agencies should be aware of the 
obligations in 19 U.S.C. § 2532, which provides that “[n]o Federal agency may engage in any 
standards-related activity that creates unnecessary obstacles to the foreign commerce of the 
United States...” and that “[e]ach Federal agency shall ensure, in applying standards-related 
activities with respect to any imported product, that such product is treated no less favorably 
than are like domestic or imported products...” Additionally, “[e]ach Federal agency shall, with 
respect to any conformity assessment procedure used by it, permit access for obtaining an 
assessment of conformity and the mark of the system, if any, to foreign suppliers of a product on 
the same basis as access is permitted to suppliers of like products, whether of domestic or other 
foreign origin.” 19 U.S.C. § 2532(4). Conformity assessment requirements and procedures are 
also subject to the WTO TBT Agreement, which requires the United States to ensure it does not 
prepare, adopt, or apply conformity assessment procedures with a view to, or with the effect of, 
creating unnecessary obstacles to international trade. Articles 5 through 9 of the WTO TBT 
Agreement set out detailed obligations on conformity assessment procedures falling within the 
scope of the WTO TBT Agreement, including the need to use relevant guides or 
recommendations issued by international standardizing bodies as a basis for conformity 
assessment procedures, except where ineffective or inappropriate, and to ensure that the 
conformity assessment procedures are not more strict or applied more strictly than necessary to 
give adequate assurance that products conform to applicable “technical regulations” or 


6 Pursuant to 15 C.F.R. § 287.1, your agency should work with NIST to “coordinate conformity assessment activities 
with those of other appropriate government agencies and with those of the private sector to reduce unnecessary 
duplication... .This will help ensure more productive use of the increasingly limited Federal resources available to 
conduct conformity assessment activities...” Consistent with 15 C.F.R. § 287.4, agencies should also “consider using 
the results of other agencies’ conformity assessment procedures (see also Section 7e).” 
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standards, taking into account the risks non-conformity would create. Article 9.1 of the WTO 
TBT Agreement provides: 

Where a positive assurance of conformity with a technical regulation or standard is required, 
Members shall, whenever practical, formulate and adopt international systems for 
conformity assessment and become members thereof or participate therein. 

The United States has also undertaken additional commitments on conformity assessment in 
U.S. free trade agreements (FTA). These include a commitment not to discriminate against 
conformity assessment bodies located in the territories of U.S. FTA partners when it comes to 
accrediting, approving, licensing or otherwise recognizing conformity assessment bodies. 
Agencies are urged to consult with USTR on questions regarding international obligations 
regarding conformity assessment. 

c. What obligations does my agency have when considering whether to use a conformity 
assessment procedure used by a trading partner? Agencies should use voluntary consensus 
standards and international guides and recommendations issued by international standardizing 
bodies in their conformity assessment procedures, as described in this Circular, including where 
such standards, guides and recommendations are used by a trading partner. In addition, Article 
6.1 of the WTO TBT Agreement obligates the United States (and other WTO members) to 
accept the results of conformity assessment procedures in other WTO member countries, 
provided it is satisfied that those procedures offer an assurance of conformity equivalent to its 
own procedures. U.S. free trade agreements may also have obligations with respect to 
recognition of trading partners’ conformity assessment procedures. Agencies should consult 
with USTR regarding compliance with these obligations. Agencies should consult with the 
Department of State in the case of other international obligations or to consider what other 
relevant international agreements may apply. Agencies should also examine how such use or 
recognition would impact the policy objectives set out in Executive Order 13609, “Promoting 
International Regulatory Cooperation.” An agency may have additional obligations to consider 
such procedures under Section 3(d) of Executive Order 13609. 

d. How does this policy affect my agency's regulatory authorities and responsibilities? This 
policy does not preempt or restrict agencies’ authorities and responsibilities to make regulatory 
decisions authorized by statute. 

e. When should my agency utilize the retrospective review mechanism with respect to 
conformity assessment? Pursuant to Executive Orders 13563, “Improving Regulation and 
Regulatory Review,” and 13610, “Identifying and Reducing Regulatory Burdens,” and 15 
C.F.R. § § 287.1 and 287.4, your agency should integrate meaningful review of its conformity 
assessment activities into its retrospective review plans on a periodic basis. Consistent with the 
NTTAA, this Circular (particularly Section 1), and NIST guidance, the review should evaluate 
alternative approaches to ensure that current conformity assessment schemes are effective and 
the burden on regulated entities is minimized. 

Agencies are encouraged to consider joint rulemaking to reduce redundancy, complexity, and 
the cumulative effects on the regulated public of needing to comply with different conformity 
assessment procedures where (1) agencies have the same or similar requirements for 
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conformity assessment for the same product/service attributes and (2) demonstrating 
compliance with one agency’s requirement is sufficient to demonstrate that a product or service 
attribute conforms to other agencies’ requirements for the same product or service attribute. 
Consistent with law and the need for agencies to maintain appropriate levels of protection, for 
example, in regulations affecting health, safety, and the environment, agencies should engage 
the regulated public and other stakeholders in public consultation as part of the retrospective 
review process described above before initiating such rulemakings. 

8. How will the U.S. Government Coordinate Regulatory Policy with Standards and 

Conformity Assessment Policy? 

The Interagency Committee on Standards Policy (ICSP) shall coordinate with the Trade Policy 
Staff Committee (TPSC), its subcommittee on Technical Barriers to Trade, and the designees, or 
“Seconds,” of the Regulatory Working Group (Working Group) created by Executive Order 
12866, “Regulatory Planning and Review,” with a view to encouraging more strategic and 
coordinated Federal participation in the development and use of standards and in conformity 
assessment activities, in accordance with the objectives of the executive orders set out in Section 
1 of this Circular. USTR has responsibility under 19 U.S.C. §§2171 and 2541 for coordinating 
the consideration of international trade policy issues with respect to standards-related activities. 

While a primary focus of this Circular is to encourage agency reliance on voluntary consensus 
standards, OMB recognizes Federal agencies also participate in the development of other 
standards with standards bodies that do not have all of the attributes described in Section 2(e), 
as well as in regulatory collaboration initiatives and encourages such participation. Under 
Section 2(a) of Executive Order 13609, “Promoting International Regulatory Cooperation,” and 
subject to Section 6 of that order, the Working Group serves “as a forum to discuss, coordinate, 
and develop a common understanding among agencies of U.S. Government positions and 
priorities with respect to: (A) international regulatory cooperation activities that are reasonably 
anticipated to lead to significant regulatory actions;...” Moreover, under Section 3(a) of 
Executive Order 13609, “if required to submit a Regulatory Plan pursuant to Executive Order 
12866, [each agency shall] include in that plan a summary of its international regulatory 
cooperation activities that are reasonably anticipated to lead to significant regulations, with an 
explanation of how these activities advance the purposes of Executive Order 13563 and 
[Executive Order 13609].” Regular discussions between the Working Group Seconds and the 
ICSP, as well as the TPSC, its subcommittee on Technical Barriers to Trade, and any other 
relevant interagency groups or committees, will help to coordinate U.S. Government activities 
relating to standards and conformity assessment, as well as regulation. 


MANAGEMENT AND REPORTING OF STANDARDS USE 


9. How Does My Agency Manage and Report on the Implementation of the Circular? 

Your agency should establish a process to identify, manage, and review your agency's activities 
pursuant to the guidance in this Circular. At minimum, your agency must have the ability to 
provide to OMB, through NIST, (1) a report on the agency's use of government-unique standards 
in lieu of voluntary consensus standards, along with an explanation of the reasons for such 
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usage, as required by Section 12(d) of the NTTAA and as described in Section 5c of this 
Circular; and (2) a summary of your agency’s activities undertaken to carry out the provisions of 
this Circular. The summary should contain a link to the agency’s standards-specific website(s) 
where this information is available. This policy establishes two ways - category based reporting 
and transaction based reporting - for agencies to manage and report their use of standards. Your 
agency must report uses of government-unique standards in lieu of voluntary consensus 
standards in one or both ways, pursuant to the following guidance. 

As required by the NTTAA, your agency must report to NIST, no later than December 31 of 
each year, the decisions by your agency in the previous fiscal year to use government-unique 
standards in lieu of voluntary consensus standards. Your agency must include an explanation of 
the reason(s) why use of such voluntary consensus standards would be inconsistent with 
applicable law or otherwise impractical, as described in Sections 5c, 10a(ii), and 1 lb(ii) of this 
Circular. Your agency must report in accordance with the instructions issued by NIST. 

NIST must maintain the reports received on http://www.nist.gov/standardsgov/ and will report 
the use of government-unique standards in lieu of voluntary consensus standards, along with the 
agency’s explanations for such use, to OMB on an annual basis. 

10. What are the Procedures for Reporting My Agency’s Use of Standards in Regulations? 

Your agency should use transaction based reporting - that is, report on each use of a 
government-unique standard in lieu of a voluntary consensus standard - in your annual report to 
NIST if your agency issues regulations that use or reference standards. If your agency is issuing 
or revising a regulation that contains a standard, your agency should follow these procedures: 

a. Publish in the preamble of rulemaking notices what steps the agency is taking to follow and 
implement the Circular. Such information should include the following elements if 
applicable: 

(i) When your agency is proposing or has decided to use a standard, provide a statement 
which identifies such standard and any alternative standards which have been 
identified. 

(ii) When your agency is proposing or has decided to use a government-unique standard in 
lieu of a voluntary consensus standard, provide a statement that identifies such 
standards and provide an explanation for why using the voluntary consensus standard 
would be inconsistent with law or otherwise impractical. Such explanation must be 
also included in the annual report. 

(iii) When your agency is proposing or has decided to use a government-unique standard 
and has not identified a voluntary consensus standard, your agency should include a 
statement to that effect in the preamble of the final rule or guidance document. For 
other rulemaking notices, your agency should invite the public to identify any standard 
and explain why your agency should use such a standard. 


b. For significant regulatory actions, agencies are also encouraged to include in their 
rulemaking notices a discussion in the regulatory preamble of the following elements: 
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(i) which bodies or organizations the agency consulted with to determine whether there 
are relevant standards in use in the marketplace (or completion of relevant standards is 
imminent) or if standards that are currently incorporated by reference have been 
revised; 

(ii) whether agency officials are participating in the work of such bodies or organizations, 
along with a description of their roles; 

(iii) whether the agency has worked with relevant bodies or organizations to try to ensure 
that the standards meet agency needs, and a description of the agency’s interactions 
with the technical committees and/or technical advisory groups involved; and 

(iv) whether the agency has coordinated its positions in technical advisory groups or 
technical committees of such bodies or organizations with (1) other interested agencies 
that are or should be participating in such work and, (2) where appropriate, foreign 
regulatory agencies that are participating in such work, where the work is relevant to 
regulatory cooperation council work plans described in Section 3(d) of Executive 
Order 13609, “Promoting International Regulatory Cooperation.” 

11. What are the Procedures for Reporting My Agency's Use of Standards in Procurements? 

Your agency must report on either a categorical basis or on a transaction basis to identify, 

manage, and review the standards used in your agency's procurements. 

a. How does my agency report the use of standards in procurements on a categorical 
basis? Your agency must use categorical based reporting when your agency identifies, 
manages, and reviews the use of standards by group or category. Category based reporting 
is especially useful when your agency either conducts large procurements or large numbers 
of procurements using government-unique standards, or is involved in long-term 
procurement contracts that require replacement parts based on government-unique 
standards. To report use of government-unique standards on a categorical basis, your 
agency must: 

(i) maintain a centralized standards management system that identifies how your agency 
uses government-unique, voluntary consensus, and other standards; 

(ii) systematically review your agency's use of government-unique standards for 
conversion to voluntary consensus standards and, where appropriate, other standards; 

(iii) maintain records on the groups or categories for which your agency uses government- 
unique standards in lieu of voluntary consensus standards, including an explanation of 
the reasons for such use, which must be transmitted according to Sections 5c and 9; 
and 

(iv) enable potential offerors to suggest voluntary consensus standards and, where 
appropriate, other standards that can replace government-unique standards. 

b. How does my agency report the use of standards in procurements on a transaction 
basis? Your agency should report on a transaction basis when your agency identifies, 
manages, and reviews the use of standards on a transaction basis rather than a category 
basis. Transaction-based reporting is especially useful when your agency conducts 
procurement involving mostly commercial products and services, but is occasionally 
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involved in a procurement involving government-unique standards. To report use of 
government-unique standards on a transaction basis, your agency must follow the following 
procedures: 

(i) in each solicitation that references government-unique standards, the solicitation must 
identify such standards and provide potential offerors an opportunity to suggest 
alternative voluntary consensus standards (or, where appropriate, other standards) that 
meet the agency's requirements; and 

(ii) if such suggestions are made and the agency decides to use government-unique 
standards in lieu of voluntary consensus standards, the agency must explain in its 
annual report to OMB through NIST, as described in Sections 5c and 9, why using 
such voluntary consensus standards is inconsistent with applicable law or otherwise 
impractical. 

The requirements in this Section do not apply to those solicitations that are for: (1) 
commercial-off-the-shelf products (COTS), (2) products or services that rely on voluntary 
consensus standards or other standards, or (3) products that otherwise do not rely on 
government-unique standards. 


AGENCY RESPONSIBILITIES 


12. What are the Responsibilities of the Secretary of Commerce? 

The Secretary of Commerce: 

a. coordinates and fosters executive branch implementation of this Circular and, as 
appropriate, provides administrative guidance to assist agencies in implementing this 
Circular, including guidance on identifying voluntary consensus standards bodies and 
voluntary consensus standards; 

b. sponsors and supports the ICSP, chaired by NIST, which considers agency views and 
advises the Secretary and agency heads on the Circular; 

c. reports to the Director of OMB concerning the implementation of the policy provisions of 
this Circular; and 

d. issues guidance to the agencies to improve coordination on conformity assessment in 
accordance with Section 7 of this Circular. 

13. What are the Responsibilities of the Heads of Agencies? 

The Heads of Agencies: 

a. implement the policies of this Circular in accordance with the procedures described; 

b. ensure agency compliance with the policies of the Circular is a top priority and implement 
appropriate organizational changes as necessary to achieve such compliance; 

c. in the case of an agency that uses standards for regulatory, procurement, or other mission- 
related activities, designate a senior level official as the Agency Standards Executive who 
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will be responsible for the agency's implementation of this Circular, including the 
responsibilities outlined in 15 C.F.R. 287.5; represent the agency on the ICSP; and be 
situated in the agency’s organizational structure such that he or she is kept regularly apprised 
of the agency’s regulatory, procurement, and other mission-related activities and has 
sufficient authority within the agency to ensure compliance with the Circular; and 

d. transmit the annual report prepared by the Agency Standards Executive as described in 
Section 9 of this Circular. 

14, What are the Qualifications and Responsibilities of Agency Standards Executives? 

An Agency Standards Executive, consistent with applicable law: 

a. Should be a senior level official; have demonstrated knowledge of, and experience in, the 
areas of standards, standardization processes and procedures, and conformity assessment; be 
broadly engaged in the agency’s standards and conformity assessment related activities so as 
to ensure intra-agency coordination; and be broadly familiar with how the agency uses 
standards and conformity assessment in its mission and how it participates in activities of 
standards bodies and conformity assessment systems, including international systems of 
conformity assessment. 

b. Promotes the following goals in the context of agency participation in the work of standards 
bodies and in conformity assessment activities: 

(i) effective use of agency resources and participation in standards bodies; 

(ii) the development of agency positions that are in the public interest and are coordinated 
across units; 

(iii) the development of agency positions that are consistent with Administration policy; 

(iv) the development of agency technical and policy positions that are clearly defined and 
coordinated with other Federal participants in a given standards activity; and 

(v) the development of agency policies on conformity assessment to implement the 
provisions of Section 7 of this Circular, consistent with 15 C.F.R. § 287. 

c. Coordinates the agency’s participation in standards bodies by: 

(i) establishing procedures to ensure that agency representatives who participate in 
standards bodies will, to the extent possible, ascertain the views of the agency on 
matters of paramount interest and express views and positions that are not inconsistent 
or in conflict with established agency views and positions; 

(ii) consulting other relevant agencies, as appropriate, to avoid to the extent practicable 
expressing views or positions in standards bodies that are inconsistent or in conflict 
with established views or positions of such other agencies; 

(iii) ensuring that the agency's participation in standards bodies is consistent with agency 
missions, authorities, priorities, and budget resources; 

(iv) coordinating with standards bodies, and notifying such bodies when the agency has 
incorporated their standards by reference in regulation; 

(v) ensuring, when two or more agencies participate in a given standards activity, that they 
coordinate their views on matters of paramount importance so as to present, whenever 
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feasible, a single, unified position and, where not feasible, a mutual recognition of 
differences; 

(vi) cooperating with the head of the agency or his or her designee in carrying out his or 
her responsibilities under this Circular; 

(vii) consulting with the head of the agency or his or her designee, as necessary, in the 
development and issuance of internal agency procedures and guidance implementing 
this Circular; 

(viii) preparing, as described in Sections 9 through 11 of this Circular, a report on uses of 
government-unique standards in lieu of voluntary consensus standards, and a report 
on the status of agency standards policy activities; 

(ix) establishing a process for ongoing review of the agency’s use of standards for 
purposes of updating or substituting, pursuant to Section 5m; 

(x) coordinating with appropriate agency offices ( e.g ., budget and legal offices) to ensure 
that effective processes exist for the review of proposed agency support for, and 
participation in, standards bodies, so that agency support and participation will comply 
with applicable laws and regulations; 

(xi) making agency employees aware of available standards training opportunities and 
providing training (consistent with an agency’s budgetary and other considerations) for 
agency employees participating in standards and conformity assessment activities so 
that participation is effective, efficient, and in keeping with the rules of the 
organizations in which they are participating; 

(xii) providing ongoing assistance and policy guidance to agency employees and managers 
on significant issues in standardization and conformity assessment; and 

(xiii) coordinating with the TPSC TBT Subcommittee or other relevant TPSC 
subcommittees on the trade implications of standardization. 

d. Ensures that agency rulemaking, procurement, and other programmatic activities are 

consistent with this Circular by, for example: 

(i) providing ongoing assistance, training, and policy guidance to agency employees and 
managers on the Circular; 

(ii) being engaged in the development of the agency’s rulemakings, procurements, and 
other programmatic activities that have a standards and/or conformity assessment 
component and providing advice on compliance with the Circular; and 

(iii) participating in the ICSP and consulting on a regular basis with NIST, other Agency 
Standards Executives, USTR (and, if applicable, your agency’s representatives to the 
relevant subcommittee of the TPSC), and the desk officer for your agency in OMB’s 
OIRA. 


SUPPLEMENTARY INFORMATION 


15, When Will This Circular Be Reviewed? 


This Circular will be reviewed for effectiveness no later than five years from the date of 
issuance. 
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16. What Is the Legal Effect of This Circular? 

This Circular is not intended to delay the administrative process, and this Circular does not 
provide new grounds for judicial review, or create new rights or benefits, substantive or 
procedural, enforceable at law or equity by a party against the United States, its agencies or 
instrumentalities, or its officers or employees. Authority for this Circular is based on 31 U.S.C. 

§ 1111, which gives OMB broad authority to establish policies for the improved management of 
the Executive Branch. This Circular is intended to implement section 12(d) of the NTTAA (15 
U.S.C. § 272 note) and other instruments set out in Section 1 of the Circular; establish policies 
that will improve the internal management of the Executive Branch; and, in so doing, improve 
regulatory and other policy outcomes. 

17. Do You Have Further Questions? 

For information concerning this Circular, contact the Office of Management and Budget, Office 
of Information and Regulatory Affairs at CircularA-119@omb.eop.gov. 
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ANNEXA 


Decision Of The Committee On Principles For The Development Of International 
Standards, Guides And Recommendations With Relation To Articles 2, 5 And Annex 3 
Of The Agreement 

Decision 7 

The following principles and procedures should be observed, when international standards, guides 
and recommendations (as mentioned under Articles 2, 5 and Annex 3 of the TBT Agreement for the 
preparation of mandatory technical regulations, conformity assessment procedures and voluntary 
standards) are elaborated, to ensure transparency, openness, impartiality and consensus, 
effectiveness and relevance, coherence, and to address the concerns of developing countries. 

The same principles should also be observed when technical work or a part of the international 
standard development is delegated under agreements or contracts by international standardizing 
bodies to other relevant organizations, including regional bodies. 


Transparency 

All essential information regarding current work programmes, as well as on proposals for standards, 
guides and recommendations under consideration and on the final results should be made easily 
accessible to at least all interested parties in the territories of at least all WTO Members. Procedures 
should be established so that adequate time and opportunities are provided for written comments. 
The information on these procedures should be effectively disseminated. 

In providing the essential information, the transparency procedures should, at a minimum, include: 

(a) The publication of a notice at an early appropriate stage, in such a manner as to enable 
interested parties to become acquainted with it, that the international standardizing body 
proposes to develop a particular standard; 

(b) The notification or other communication through established mechanisms to 
members of the international standardizing body, providing a brief description of the 
scope of the draft standard, including its objective and rationale. Such 
co mm unications shall take place at an early appropriate stage, when amendments can 
still be introduced and comments taken into account; 

(c) Upon request, the prompt provision to members of the international standardizing body 
of the text of the draft standard; 

(d) The provision of an adequate period of time for interested parties in the territory of at 
least all members of the international standardizing body to make co mm ents in writing 


7 G/TBT/9, 13 November 2000, para. 20 and An nex 4. 
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and take these written comments into account in the further consideration of the 
standard; 

(e) The prompt publication of a standard upon adoption; and 

(f) To publish periodically a work programme containing information on the standards 
currently being prepared and adopted. 

It is recognized that the publication and communication of notices, notifications, draft standards, 
comments, adopted standards or work programmes electronically, via the Internet, where feasible, 
can provide a useful means of ensuring the timely provision of information. At the same time, it is 
also recognized that the requisite technical means may not be available in some cases, particularly 
with regard to developing countries. Accordingly, it is important that procedures are in place to 
enable hard copies of such documents to be made available upon request. 

Openness 

Membership of an international standardizing body should be open on a non- discriminatory basis to 
relevant bodies of at least all WTO Members. This would include openness without discrimination 
with respect to the participation at the policy development level and at every stage of standards 
development, such as the: 

(a) Proposal and acceptance of new work items; 

(b) Technical discussion on proposals; 

(c) Submission of comments on drafts in order that they can be taken into account; 

(d) Reviewing existing standards; 

(e) Voting and adoption of standards; and 

(f) Dissemination of the adopted standards. 

Any interested member of the international standardizing body, including especially 
developing country Members, with an interest in a specific standardization activity should be 
provided with meaningful opportunities to participate at all stages of standard development. It is 
noted that with respect to standardizing bodies within the territory of a WTO Member that have 
accepted the Code of Good Practice for the Preparation, Adoption and Application of Standards by 
Standardizing Bodies (Annex 3 of the TBT Agreement) participation in a particular international 
standardization activity takes place, wherever possible, through one delegation representing all 
standardizing bodies in the territory that have adopted, or expected to adopt, standards for the 
subject-matter to which the international standardization activity relates. This is illustrative of the 
importance of participation in the international standardizing process accommodating all relevant 
interests. 
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Impartiality and Consensus 

All relevant bodies of WTO Members should be provided with meaningful opportunities to 
contribute to the elaboration of an international standard so that the standard development process 
will not give privilege to, or favour the interests of, a particular supplier/s, country/ies or region/s. 
Consensus procedures should be established that seek to take into account the views of all parties 
concerned and to reconcile any conflicting arguments. 

Impartiality should be accorded throughout all the standards development process with respect to, 
among other things: 

(a) Access to participation in work; 

(b) Submission of comments on drafts; 

(c) Consideration of views expressed and comments made; 

(d) Decision-making through consensus; 

(e) Obtaining of information and documents; 

(f) Dissemination of the international standard; 

(g) Fees charged for documents; 

(h) Right to transpose the international standard into a regional or national standard; and 

(i) Revision of the international standard. 

Effectiveness and Relevance 

In order to serve the interests of the WTO membership in facilitating international trade and 
preventing unnecessary trade barriers, international standards need to be relevant and to effectively 
respond to regulatory and market needs, as well as scientific and technological developments in 
various countries. They should not distort the global market, have adverse effects on fair 
competition, or stifle innovation and technological development. In addition, they should not give 
preference to the characteristics or requirements of specific countries or regions when different 
needs or interests exist in other countries or regions. Whenever possible, international standards 
should be performance based rather than based on design or descriptive characteristics. 

Accordingly, it is important that international standardizing bodies: 

(a) Take account of relevant regulatory or market needs, as feasible and appropriate, as well 
as scientific and technological developments in the elaboration of standards; 

(b) Put in place procedures aimed at identifying and reviewing standards that have become 
obsolete, inappropriate or ineffective for various reasons; and 

(c) Put in place procedures aimed at improving communication with the World Trade 
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Organization. 

Coherence 

In order to avoid the development of conflicting international standards, it is important that 
international standardizing bodies avoid duplication of, or overlap with, the work of other 
international standardizing bodies. In this respect, cooperation and coordination with other relevant 
international bodies is essential. 

Development Dimension 

Constraints on developing countries, in particular, to effectively participate in standards 
development, should be taken into consideration in the standards development process. Tangible 
ways of facilitating developing countries' participation in international standards development 
should be sought. The impartiality and openness of any international standardization process 
requires that developing countries are not excluded de facto from the process. With respect to 
improving participation by developing countries, it may be appropriate to use technical assistance, 
in line with Article 11 of the TBT Agreement. Provisions for capacity building and technical 
assistance within international standardizing bodies are important in this context. 
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UNITED STATES DISTRICT COURT 
FOR THE DISTRICT OF COLUMBIA 


) 

AMERICAN SOCIETY FOR TESTING ) 

AND MATERIALS, et al., ) 

) 

Plaintiffs, ) 

) Case No. 13-cv-1215 (TSC) 
v. ) 

) 

PUBLIC.RESOURCE.ORG, INC., ) 

) 

Defendant. ) 

_ » 

ORDER 

Defendant Public Resource moves to strike the expert report of John C. Jarosz (“Jarosz 
Report”) (ECF No. 118-12, Ex. 1) on the basis that it does not meet the requirements of Federal 
Rule of Evidence 702. The Jarosz Report is used primarily to support Plaintiffs’ economic 
arguments regarding the harm to their revenue and incentives if the court were to find that 
incorporation of their standards by reference into federal regulations revokes or destroys their 
copyrights, or Defendant was otherwise allowed to continue posting the standards on its website. 
For the reasons stated herein, Defendant’s motion is DENIED. 

A district court has ‘“broad discretion in determining whether to admit or exclude expert 
testimony.’” United States ex rel. Miller v. Bill Harbert Int’l Constr., Inc., 608 F.3d 871, 895 
(D.C. Cir. 2010) (quoting United States v. Gatling, 96 F.3d 1511, 1523 (D.C. Cir. 1996)). Under 
the Supreme Court’s decision in Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 
(1993), this court is “required to address two questions, first whether the expert’s testimony is 
based on ‘scientific knowledge’ and second, whether the testimony ‘will assist the trier of fact to 
understand or determine a fact in issue.’” Meister v. Med. Eng’g Corp., 267 F.3d 1123, 1126 
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(D.C. Cir. 2001) (quoting Daubert, 509 U.S. at 592). Trial courts “act as gatekeepers who may 
only admit expert testimony if it is both relevant and reliable,” Heller v. D.C., 952 F. Supp. 2d 
133, 139 (D.D.C. 2013), though this role is “significantly diminished” at the summary judgment 
stage, see Window Specialists, Inc. v. Forney Enters., Inc., 47 F. Supp. 3d 53, 60 (D.D.C. 2014). 

In determining whether to strike an expert report, the court’s focus is on whether the 
expert’s assumptions “amount to ‘rampant speculation’ and should be excluded” or “merely 
represent a weak factual basis for his testimony” which could be appropriately challenged on 
cross examination at trial. Boyar v. Korean Air Lines Co., Ltd., 954 F. Supp. 4, 7 (D.D.C. 1996). 

As the Court in Daubert instructed, “vigorous cross examination, presentation of contrary 
evidence, and careful instruction on the burden of proof are the traditional and appropriate means 
of attacking shaky but admissible evidence.” 509 U.S. at 596. 

Defendant argues that the scope of the Jarosz Report exceeds his expertise and that Jarosz 
improperly relied on factual information from Plaintiffs themselves, thus acting “as a 
mouthpiece.” (Def. Mem. at 6-7). Based on Jarosz’s education, publications, and participation 
as an expert in intellectual property infringement in hundreds of other cases, the court finds his 
expertise to be well established. While Defendant argues that Jarosz lacks both experience 
evaluating standards development organizations and independent knowledge of the development 
of those organizations’ standards and the process of incorporation by reference, the court 
concludes that such specialized personal knowledge is not required for an expert to be qualified 
to opine on the economic impact of copyright infringement. Additionally, based on the extensive 
number of deposition transcripts, documents, websites, publications, and data reviewed by 
Jarosz, his opinions are sufficiently supported. 

Defendant also argues that Jarosz made improper assumptions and failed to apply reliable 
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methodologies to the facts. Specifically, Defendant takes issue with Jarosz’s analysis involving 
the impact on revenue from the loss of copyright protection, the differences in harms relating to 
the standards in this case versus all of Plaintiffs’ standards generally, the potential impact that 
Plaintiffs’ reading rooms have on revenue, and the estimation of lost sales. Ultimately, 

Defendant appears to argue simply that different analyses would have resulted in an expert report 
more favorable to Defendant’s position. Defendant could have offered a rebuttal expert in 
response (and was in fact given time to do so by Magistrate Judge Robinson during discovery), 
but chose not to. However, the court will not strike an expert report simply because the expert 
did not rely on the particular assumptions or data Defendant thought was necessary. Those 
issues are more properly addressed through “vigorous cross-examination [and] presentation of 
contrary evidence.” Daubert, 509 U.S. at 596. 

Plaintiffs have sufficiently established that Jarosz has the experience and education 
necessary to be qualified as an expert in this case, and that the content of his testimony— 
applying general economic principles to the effects of copyright infringement of Plaintiffs’ 
standards—may “help the trier of fact.” See Fed. R. Evid. 702; Daubert, 509 U.S. at 588. 
Therefore, at this stage, the court will not take the unusual step of striking his report from 
consideration. 

Date: September 21,2016 

TANYA S. CHUTKAN 
United States District Judge 
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I mean, it is not valid unless an instrument of conveyance or a 
note or memorandum of the transfer is in writing and signed by 
the owner of the rights conveyed or such owner's duly authorized 
agent. And the cases are clear that when you say on these 
membership forms, oh, I agree that anything I do will belong to 
you, that's not an assignment. So that's the ASTM problem. 

It's a severe problem. 

Then we get to NFPA, and I will admit that the most recent 
NFPA standard is better. Okay? It is absolutely better. 

That's why they amended the complaint to add it to the lawsuit, 
because it may be the only document at issue in this case where 
there looks like pretty good ownership. But even there, there's 
a problem, Your Honor, and this gets a little technical. 

Now that they claim that everything is joint works from 
joint owners, what about the fact that some of these joint 
owners are the U.S. government? That U.S. government employees 
participate as joint authors? 

No case has ever dealt with this. Your Honor, and I don't 
know how to deal with it. But § 105 of the Copyright Act says 
that U.S. government works are not subject to copyright, and 
Mr. Klaus explained that those are, where they're prepared by an 
employee acting in the scope of employment. Now they're saying 
they've got joint works with a whole bunch of federal employees 
as joint authors. 

So this is just a mess. Your Honor, yes. It is a dog's 
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THE COURT: Thank you. 

MR. BRIDGES: Thank you, Your Honor. This is 
Andrew Bridges again for Public.Resource. 

THE COURT: Mr. Bridges, if the defendant's sole 
purpose is to disseminate the law, as you say, why do you need 
to disseminate the plaintiffs' logos? 

MR. BRIDGES: We don't have to. Your Honor, except 
that what we've done is, in the spirit of what we understand the 
incorporation is to be, which is incorporation of particular 
documents, Public.Resource has replicated the entire document. 

As is. Now, we need — 

THE COURT: Well, then you add this certificate; right? 

MR. BRIDGES: That's right, which emphatically 
makes the point that it is the law. It doesn't say this is 
Public.Resource's. We need to be clear. The allegations that 
Public.Resource is trying to confuse the public about source 
sponsorship or affiliation of these standards is pretextual and 
ironic. The fact is, they would sue Public.Resource no matter 
what. If Public.Resource dropped the logos, they would sue for 
reverse passing-off, but because it maintained the logos, 
they're suing for trademark infringement. 

Let me be clear. Public.Resource would take direction from 
this Court. Logos: yes or no? It doesn't care. It simply 
tried to replicate the law which consists of these documents 
incorporated by reference. 


JA2056 



1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 


Case l:13-cv-01215-TSC Document 173 Filed 10/13/16 Page 117 of 142 
Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 298 of 573 W 


Disclaimer. First of all, the Supreme Court in two cases 
has approved disclaimers. If Public.Resource needs to say — 
first of all, I'm not sure that the plaintiffs would want their 
logos taken off because they use their monopoly position to try 
to make money by associating these standards that have become 
law with themselves. But if they want the logos off, we will 
get the logos off. Your Honor. That's not a sticking point. 
We're just trying to make clear that these are the laws that are 
in the CFR or state law or whatever. If the Court wants a 
disclaimer — 

THE COURT: Well, with regard to disclaimer, if you 
point to your disclaimers as sufficient to notify consumers that 
the standards aren't originals, that they're reproductions, I 
look at the language on the cover page, and it's hard to 
understand how this — is this Exhibit 16? — how this resolves 
any confusion. 

MR. BRIDGES: Your Honor, it's not just about this. 

It's about the entire experience that somebody has going to 
Public.Resource's website. When I go to the Cornell website, I 
don't think I'm going to the Library of Congress to get a law. 

I know I'm going someplace where I can get the law. I've got no 
confusion between the National Archives and Cornell, but I know 
that I can go to Cornell to get the law. There is no likelihood 
of confusion that somebody thinks Public.Resource wrote these. 

THE COURT: Then why do you have a disclaimer? 
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MR. BRIDGES: We have this document that says this is 
the law. We have — and I'm not — there are different 
disclaimers at different times, so I'm not clear on exactly what 
they've all been. 

THE COURT: Why do you even need this? 

MR. BRIDGES: We need this to make a political point 
that this is the law, and we want people to understand that this 
is no longer just somebody's private standard. This is the law, 
and that's exactly what it says here. It's giving the citation 
to the U.S. Code that makes it the law. 

THE COURT: If all you want to do is to make sure that 
consumers realize that it is the law, why do you need their logo? 

MR. BRIDGES: I'm saying. Your Honor, we would drop 
the logo in a second if that's the Court's direction. The 
reason we included the logo — we don't have to have a fight 
over them with this. 

THE COURT: Well, they brought a claim. 

MR. BRIDGES: That's right. They brought a claim, and 
they would have brought a claim no matter what we did, because 
it's really a copyright issue. 

THE COURT: The Court is unconcerned with their 
motivations for bringing a claim. My only concern is whether 
they have a valid claim. 

MR. BRIDGES: Your Honor, if the motivation is to 
enforce a copyright right, then it's squarely in the middle of 
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UNITED STATES DISTRICT COURT 
FOR THE DISTRICT OF COLUMBIA 


) 

AMERICAN SOCIETY FOR TESTING ) 

AND MATERIALS, et al., ) 

) 

Plaintiffs, ) 

) Case No. 13-cv-1215 (TSC) 
v. ) 

) 

PUBLIC.RESOURCE.ORG, INC., ) 

) 

Defendant. ) 

) 

AMERICAN EDUCATIONAL ) 

RESEARCH ASSOCIATION, INC. et al, ) 

) 

Plaintiffs, ) 

) 

v. ) Case No. 14-cv-0857 (TSC) 

) 

PUBLIC.RESOURCE.ORG, INC., ) 

) 

Defendant. ) 

_ ) 

MEMORANDUM OPINION 


Before the court are motions and cross-motions for summary judgment in two related 
cases. Because there is significant factual and legal overlap between the two cases, the court 
issues this consolidated opinion to be filed in both cases. 

Plaintiffs American Society for Testing and Materials (“ASTM”), National Fire 
Protection Association, Inc. (“NFPA”), and American Society of Heating, Refrigerating, and 
Air-Conditioning Engineers (“ASHRAE”) (collectively “ASTM Plaintiffs”) brought suit against 
Defendant Public.Resource.org, Inc. (“Public Resource”) under the Copyright Act (17 U.S.C. 

§ 101 etseq .) and the Lanham Act (15 U.S.C. § 1051 et seq.), alleging copyright infringement 
and trademark infringement. Plaintiffs American Educational Research Association, Inc. 
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(“AERA”), American Psychological Association, Inc. (“APA”), and National Council on 
Measurement in Education, Inc. (“NCME”) (collectively “AERA Plaintiffs”) also brought 
copyright infringement claims against Public Resource under the Copyright Act. Plaintiffs 1 in 
both cases seek permanent injunctions barring Defendant from continued display of their works. 

Plaintiffs moved for summary judgment, and Defendant filed cross-motions for summary 
judgment in both cases. The court held a combined oral argument on September 12, 2016 to 
consider the motions. Upon consideration of the parties’ filings, the numerous amicus briefs, 
and the arguments presented at the motions hearing, and for the reasons stated herein, the ASTM 
Plaintiffs’ motion for summary judgment is GRANTED and Defendant’s cross-motion is 
DENIED. The AERA Plaintiffs’ motion for summary judgment is GRANTED IN PART AND 
DENIED IN PART, and Defendant’s cross-motion is DENIED. 

I. FACTUAL BACKGROUND 

A, The Parties 

1. ASTM Plaintiffs 

ASTM Plaintiffs are not-for-profit organizations that develop private sector codes and 
standards in order to advance public safety, ensure compatibility across products and services, 
facilitate training, and spur innovation. (See ASTM Pis. Statement of Material Facts (“PSMF”) 

HI 9, 13, 14, 86, 87, 129, 130 (ASTM ECF No. 118-2)). 2 These standards include technical 
works, product specifications, installation methods, methods for manufacturing or testing 
materials, safety practices, and other best practices or guidelines. (Id. ][ I ). ASTM has 


1 For simplicity, the court’s use of “Plaintiffs” refers collectively to the ASTM Plaintiffs and 
AERA Plaintiffs. 

2 All initial citations to the record in this Opinion will include the docket number as “ASTM 
ECF” or “AERA ECF.” 
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developed over 12,000 standards that are used in a wide range of fields, including consumer 
products, iron and steel products, rubber, paints, plastics, textiles, medical services and devices, 
electronics, construction, energy, water, and petroleum products, and are the combined efforts of 
over 23,000 technical members, representing producers, users, consumers, government, and 
academia. (Id. ]fl| 13, 28, 41). NFPA has developed over 300 standards in the areas of fire, 
electrical, and building safety, with the goal of reducing the risk of death, injury, and property 
and economic loss due to fire, electrical, and related hazards. {Id. 86, 87, 92). NFPA’s most 
well-known standard is the National Electrical Code, first published in 1897 and most recently in 
2014. {Id. ]fl[ 93-94). Finally, ASHRAE has published over 100 standards for a variety of 
construction-related fields, including energy efficiency, indoor air quality, refrigeration, and 
sustainability. {Id. | 130). 

2. AERA Plaintiffs 

AERA Plaintiffs are not-for-profit organizations that collaboratively develop the 
Standards for Educational and Psychological Testing, including the 1999 edition at issue in this 
case (“the 1999 Standards”). (AERA PSMF fflf 1, 5, 13 (AERA ECF No. 60-2)). AERA is a 
national scientific society whose mission is “to advance knowledge about education, to 
encourage scholarly inquiry related to education, and to promote the use of research to improve 
education.” {Id. ][ 2). APA is the world’s largest association of psychologists, and its mission is 
“to advance the creation, communication, and application of psychological knowledge.” {Id. 

]j 3). Finally, NCME is a professional organization “for individuals involved in assessment, 
evaluation, testing, and other aspects of educational measurement.” (Id. ][ 4). 

3. Public Resource 

Defendant Public Resource is a not-for-profit entity devoted to publicly disseminating 
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legal information. (ASTM DSMF 1-2 (ASTM ECF No. 120-3); AERA DSMF ffl[ 1-2 
(AERA ECF No. 68-3)). Its mission is “make the law and other government materials more 
widely available so that people, businesses, and organizations can easily read and discuss [the] 
laws and the operations of government.” (ASTM DSMF 12; AERA DSMF 12). Public 
Resource has posted government-authored materials on its website, including judicial opinions, 
Internal Revenue Service records, patent filings, and safety regulations. (ASTM DSMF ^|f 3—4; 
AERA DSMF 3—4). It does not charge fees to view or download the materials on its website. 
(ASTM DSMF f 5; AERA DSMF f 5). 

B. Incorporation by Reference of Industry Standards 

In the United States, a complex public-private partnership has developed over the last 
century in which private industry groups or associations, rather than government agencies, 
typically develop standards, guidelines, and procedures that set the best practices in a particular 
industry. 3 Applicable standards are used by entities and individuals in order to self-regulate and 
conform to the best practices of that industry. Professor Peter Strauss has noted that 
“manufacturing and markets are greatly aided, and consumers offered protection, by the 
application of uniform industrial standards created independent of law, as means of assuring 
quality, compatibility, and other highly desired market characteristics.” Peter L. Strauss, Private 
Standards Organizations and Public Law, 22 Wm. & Mary Bill Rts. J. 497, 499 (2013). 


3 See U.S. Office of Management and Budget, Revised Circular No. A-l 19, 
https://obamawhitehouse.archives.gov/sites/default/files/omb/inforeg/revised_circular_a- 
119_as_of_l_22.pdf (“OMB Revised Circular”) at 1 (Jan. 27, 2016) (“The vibrancy and 
effectiveness of the U.S. standards system in enabling innovation depends on continued private 
sector leadership and engagement. Our approach—reliance on private sector leadership, 
supplemented by Federal government contributions to discrete standardization processes as 
outlined in OMB Circular A-l 19—remains the primary strategy for government engagement in 
standards development.”). 
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Standards are typically developed by standards developing organizations (“SDOs”), like 
Plaintiffs, who work to develop “voluntary consensus standards,” such as those here. Voluntary 
consensus standards are the ultimate product of many volunteers and association members from 
numerous sectors bringing together technical expertise. They are “developed using procedures 
whose breadth of reach and interactive characteristics resemble governmental rulemaking, with 
adoption requiring an elaborate process of development, reaching a monitored consensus among 
those responsible within the SDO.” Id. at 501. ASTM Plaintiffs develop their standards using 
technical committees with representatives from industry, government, consumers, and technical 
experts. (ASTM PSMF 7, 28, 29, 109, 114, 135). These committees conduct open 
proceedings, consider comments and suggestions, and provide for appeals, and through 
subcommittees, draft new standards, which the full committees vote on. (Id. fflf 31-37, 109, 136, 
139). The AERA Plaintiffs developed the 1999 Standards through a Joint Committee which 
considered input from the public in a notice-and-comment process. (AERA PSMF 13-16). 

Pursuant to 5 U.S.C. § 552, federal agencies may incorporate voluntary consensus 
standards—as well as, for example, state regulations, government-authored documents, and 
product service manuals—into federal regulations by reference. See Emily S. Bremer, 
Incorporation by Reference in an Open-Government Age, 36 Harv. J.L. & Pub. Pol’y 131, MS- 
47 (2013) (providing a general overview of the federal government’s incorporation of materials 
by reference). The federal government’s practice of incorporation by reference of voluntary 
consensus standards is intended to achieve several goals, including eliminating the cost to the 
federal government of developing its own standards, encouraging long-term growth for U.S. 
enterprises, promoting efficiency, competition, and trade, and furthering the reliance upon 
private sector expertise. See OMB Revised Circular, supra, at 14. 
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Section 552(a)(1) provides that “a person may not in any manner be required to resort to, 
or be adversely affected by, a matter required to be published in the Federal Register and not so 
published[, but] . . . matter reasonably available to the class of persons affected thereby is 
deemed published in the Federal Register when incorporated by reference therein with the 
approval of the Director of the Federal Register.” 5 U.S.C. § 552(a)(1) (emphasis added). The 
Office of the Federal Register (“OFR”) adopted regulations pursuant to § 552(a)(1) in 1982 and 
issued revised regulations in 2014. See Approval Procedures for Incorporation by Reference, 47 
Fed. Reg. 34,107 (Aug. 6, 1982) (codified at 1 C.F.R. § 51.1 etseq.); 79 Fed. Reg. 66,267 (Nov. 

7, 2014). These regulations specify that a “publication is eligible for incorporation by reference” 
if it is “published data, criteria, standards, specifications, techniques, illustrations, or similar 
material; and [d]oes not detract from the usefulness of the Federal Register publication system.” 

1 C.F.R. § 51.7(a)(2). To determine whether the material is “reasonably available” as required 
by the statute, OFR will consider “[t]he completeness and ease of handling of the publication” 
and “[w]hether it is bound, numbered, and organized, as applicable.” Id. § 51.7(a)(3). All the 
standards at issue in this case have been incorporated by reference into federal law. (ASTM 
DSMF122; 34 C.F.R. § 668.146 (incorporating AERA Plaintiffs’ 1999 Standards). 

Standards that are incorporated by reference are available in person at the OFR in 
Washington, DC and/or with the incorporating agency. See 1 C.F.R. § 51.3(b)(4). Federal 
regulations that incorporate standards by reference typically direct interested individuals or 
entities to location(s) where they may view the incorporated documents in person. For example, 
the Environmental Protection Agency’s (“EPA”) regulation, 40 C.F.R. § 60.17(a), which 
incorporates numerous standards at issue here, states that: 

Certain material is incorporated by reference into this part with the approval of the 

Director of the Federal Register under 5 U.S.C. § 552(a) and 1 CFR part 51. . . . 
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All approved material is available for inspection at the EPA Docket Center, Public 
Reading Room, EPA WJC West, Room 3334, 1301 Constitution Ave. NW, 
Washington, DC, telephone number 202-566-1744, and is available from the 
sources listed below. It is also available for inspection at the National Archives and 
Records Administration (NARA). For information on the availability of this 
material at NARA, call (202) 741-6030 or go to http://www.archives.gov/ 
federalregister/codeoffederalregulations/ibrlocations.html. 

The EPA regulation further specifies that, for example, the 206 ASTM standards incorporated by 

reference by the EPA (some of which are involved in this suit) are “available for purchase from 

ASTM International, 100 Barr Harbor Drive, P.O. Box CB700, West Conshohocken, 

Pennsylvania 19428-2959, (800) 262-1373, http://www.astm.org.” 40 C.F.R. § 60.17(h). The 

U.S. Department of Education incorporated the AERA Plaintiffs’ 1999 Standards by reference at 

34 C.F.R. § 668.146(b)(6), which states that the standards are: 

on file at the Department of Education, Federal Student Aid, room 113E2, 830 First 
Street, NE, Washington, DC 20002, phone (202) 377-4026, and at the National 
Archives and Records Administration (NARA). For information on the availability 
of this material at NARA, call 1-866-272-6272, or to go: http://www.archives.gov/ 
federal-register/code-of-federal-regulations/ibr-locations.html. The document may 
also be obtained from the American Educational Research Association. 

ASTM Plaintiffs sell PDF and hard copy versions of their standards, including those that 

have been incorporated by reference into law. (ASTM PSMF 57, 99, 157). The prices for the 

standards in this case range from $25 to $200. (Id. 58, 99, 158). The ASTM Plaintiffs also 

maintain “reading rooms” on their websites that allow interested parties to view Plaintiffs’ 

standards that have been incorporated by reference. (Id. 63-64, 100, 161). The standards in 

these reading rooms are “read-only,” meaning they appear as images that may not be printed or 

downloaded. (Id.). AERA Plaintiffs sell hardcopy versions of the 1999 Standards, but do not 

sell digital or PDF versions. (AERA PSMF 30, 33). The prices for the 1999 Standards have 

ranged from $25.95 to $49.95 per copy, and they were sold continuously from 2000 through 

2014, except for a nearly two-year period. (Id. 34-35). 
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C. Plaintiffs’ Claims in This Action 

1. ASTM et al. v. Public Resource 

This case involves 257 of ASTM Plaintiffs’ standards that have been incorporated by 
reference into federal law. (See ASTM Compl. Ex. A-C; ASTM DSMF ]f 22). Defendant 
admits that it purchased hard copies of each of the standards at issue, scanned them into PDF 
files, added a cover sheet, and posted them online. (ASTM DSMF fflf 173-74, 177-78; ASTM 
PSMF 182-87). Defendant re-typed some of ASTM Plaintiffs’ standards and posted them 
online, with text in Hypertext Markup Fanguage (HTMF) format and graphics and figures in 
Mathematics Markup Fanguage and Scalable Vector Graphics formats. (ASTM DSMF 83, 

175). The copies posted on Defendant’s website all bore ASTM Plaintiffs’ trademarks. (ASTM 
PSMF 1210). Defendant also uploaded the ASTM Plaintiffs’ standards to the Internet Archive, 
a separate independent website. {Id. 1185). 

The ASTM Plaintiffs allege that their standards are original works protected from 
copyright infringement, and brought claims of copyright infringement, contributory copyright 
infringement, trademark infringement, unfair competition and false designation, and trademark 
infringement under common law. (ASTM Compl. 142-95). Defendant counter-sued, seeking 
a declaratory judgment that its conduct does not violate copyright law or trademark law. (ASTM 
Ans. lfl[ 174—205). Both sides have filed motions for summary judgment. 

2. AERA et al. v. Public Resource 

This case involves the 1999 Standards, which AERA Plaintiffs have sold since 2000. 
(AERA PSMF ]j][ 34—35). In May 2012, Public Resource purchased a paper copy of the 1999 
Standards, disassembled it, scanned the pages, created a PDF file, attached a cover sheet, and, 
without authorization from the AERA Plaintiffs, posted the PDF file to Public Resource’s 
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website and the Internet Archive. (AERA DSMF ][ 28; AERA PSMF ][][ 69-80). Public 
Resource posted a read-only version of the 1999 Standards to its website, unlike many of the 
ASTM Plaintiffs’ standards, which had undergone optical character recognition (“OCR”) 
processing to be text-searchable. {Id. 173). OCR processing uses a machine to recognize letters 
and words in a PDF and translate them into letters or words that can be searched and used by 
text-to-speech software for individuals who are blind or visually impaired. (Id. ]|]| 73-75). 

Plaintiffs allege that the 1999 Standards are protected original works, and they brought 
suit claiming copyright infringement and contributory copyright infringement. (AERA Compl. 

Tff 50-63). Defendant counter-sued seeking a declaratory judgment that its conduct does not 
violate copyright law or trademark law. (AERA Ans. 116-37). Both sides have moved for 
summary judgment. 

II. LEGAL STANDARD 

Summary judgment may be granted if “the movant shows that there is no genuine dispute 
as to any material fact and the movant is entitled to judgment as a matter of law.” Fed. R. Civ. P. 
56(a); see also Anderson v. Liberty Lobby, Inc., All U.S. 242, 247^48 (1986) (“[T]he mere 
existence of some alleged factual dispute between the parties will not defeat an otherwise 
properly supported motion for summary judgment; the requirement is that there be no genuine 
issue of material fact.”) (emphasis in original); Holcomb v. Powell, 433 F.3d 889, 895 (D.C. Cir. 
2006). Summary judgment may be rendered on a “claim or defense ... or [a] part of each claim 
or defense.” Fed. R. Civ. P. 56(a). 

“A party asserting that a fact cannot be or is genuinely disputed must support the 
assertion by ... citing to particular parts of materials in the record.” Fed. R. Civ. P. 56(c)(1)(A). 

“A fact is ‘material’ if a dispute over it might affect the outcome of a suit under governing law; 
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factual disputes that are ‘irrelevant or unnecessary’ do not affect the summary judgment 
determination. An issue is ‘genuine’ if ‘the evidence is such that a reasonable jury could return a 
verdict for the nonmoving party.’” Holcomb, 433 F.3d at 895 (quoting Liberty Lobby, 477 U.S. 
at 248) (citation omitted). The party seeking summary judgment “bears the heavy burden of 
establishing that the merits of his case are so clear that expedited action is justified.” Taxpayers 
Watchdog, Inc. v. Stanley, 819 F.2d 294, 297 (D.C. Cir. 1987). 

In considering a motion for summary judgment, “[t]he evidence of the non-movant is to 
be believed, and all justifiable inferences are to be drawn in his favor.” Liberty Lobby, 477 U.S. 
at 255; see also Mastro v. Potomac Elec. Power Co., 447 F.3d 843, 850 (D.C. Cir. 2006) (“We 
view the evidence in the light most favorable to the nonmoving party and draw all inferences in 
its favor.”). The nonmoving party’s opposition, however, must consist of more than mere 
unsupported allegations or denials, and must be supported by affidavits, declarations, or other 
competent evidence setting forth specific facts showing that there is a genuine issue for trial. See 
Fed. R. Civ. P. 56(c); Celotex Corp. v. Catrett, All U.S. 317, 324 (1986). The non-movant “is 
required to provide evidence that would permit a reasonable jury to find [in his favor].” 

Laningham v. U.S. Navy, 813 F.2d 1236, 1242 (D.C. Cir. 1987). 

III. ANALYSIS 

A. Copyright Infringement 

Under the Copyright Act, copyright in a work vests initially in the author(s) of that work. 

17 U.S.C. § 201(a). Ownership can be transferred in whole or in part, and the exclusive rights of 
copyright ownership may also be transferred. Id. § 201(d). An owner of a valid copyright has 
the “exclusive right” to reproduce, distribute, or display the copyrighted works as well as prepare 
derivative works based upon it. Id. § 106(1)—(3), (5). Anyone who violates the exclusive rights 

10 


JA2068 



Case l:13-cv-01215-TSC Document 175 Filed 02/02/17 Page 11 of 55 
USCA Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 310 of 573 

of the copyright owner “is an infringer of the copyright or right of the author, as the case may 
be.” Id. § 501(a). The legal or beneficial owner of that exclusive right may then “institute an 
action for any infringement.” Id. § 501(b). In order to succeed on their copyright infringement 
claims, the Plaintiffs must prove both ‘“(1) ownership of a valid copyright, and (2) copying of 
constituent elements of the work that are original.’” Stenograph, LLC v. Bossard Assoc., Inc., 

144 F.3d 96, 99 (D.C. Cir. 1998) (quoting FeistPubl’ns, Inc. v. Rural Tel. Serv. Co., 499 U.S. 

340, 361 (1991)). 

1. Feist Prong 1: Ownership of a Valid Copyright 

a. Ownership 

The court must first decide the threshold issue of whether Plaintiffs own the copyrights in 
part or outright such that they have standing to challenge Defendant’s alleged infringement. The 
Copyright Act provides that possession of a certificate of registration from the U.S. Copyright 
Office “made before or within five years after first publication of the work shall constitute prima 
facie evidence,” creating a rebuttable presumption of ownership of a valid copyright. 17 U.S.C. 

§ 410(c); see also MOB Music Publ’g. v. Zanzibar on the Waterfront, LLC, 698 F. Supp. 2d 197, 
202 (D.D.C. 2010). If the copyright was registered more than five years after the work was 
published, then the “evidentiary weight to be accorded .. . shall be within the discretion of the 
court.” 17 U.S.C. § 410(c). 

When a party offers as prima facie evidence a registration certificate for a compilation of 
individual works that it authored, rather than the registration for a specific individual work, a 
court may consider this to be similar prima facie evidence of ownership, creating the same 
rebuttable presumption. SeeXoom, Inc. v. Imageline, Inc., 323 F.3d 279, 283-84 (4th Cir. 2003), 
abrogated by Reed Elsevier, Inc. v. Muchnick, 559 U.S. 154 (2010); Morris v. Business 
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Concepts, Inc., 259 F.3d 65, 68 (2d Cir. 2001), abrogated on other grounds by Muchnick, 559 
U.S. 154 (2010). Moreover, the registration certificate is sufficient prima facie evidence for the 
individual works within the compilation if the compilation is deemed to be a “single work.” 

Federal regulations provide that “all copyrightable elements that are otherwise recognizable as 
self-contained works, that are included in a single unit of publication, and in which the copyright 
claimant is the same” constitute a “single work,” such that they are validly registered under a 
single registration certificate 37 C.F.R. § 202.3(b)(4)(A); Kay Berry, Inc. v. Taylor Gifts, Inc., 

4221 F.3d 199, 205-06 (3d Cir. 2005); Yurman Studio, Inc. v. Castaneda, 591 F. Supp. 2d 471, 

483 (S.D.N.Y. 2008). 

Once a copyright holder has proffered this prima facie evidence, the alleged infringer 
“challenging the validity of the copyright has the burden to prove the contrary.” Hamil Am., Inc. 
v. GFI, Inc., 193 F.3d 92, 98 (2d Cir. 1999); United Fabrics Int’l, Inc. v. C&J Wear, Inc., 630 
F.3d 1255, 1257 (9th Cir. 2011) (infringer “has the burden of rebutting the facts set forth in the 
copyright certificate”). The defendant-infringer might argue that the plaintiff-copyright holder 
had some defect in the record-keeping submitted to establish ownership. However, this “skips a 
step,” as the defendant must first “set forth facts that rebut the presumption of validity to which 
[the plaintiffs] copyright is entitled” before attacking the sufficiency of a plaintiffs evidence of 
ownership. United Fabrics, 630 F.3d at 1257. The infringer must use “ other evidence in the 
record [to] cast[] doubt on” the validity of the ownership. Fonar Corp. v. Domenick, 105 F.3d 
99, 104 (2d Cir. 1997) (emphasis in original). The court in Fonar noted that defendant-infringers 
have overcome the presumption of validity with evidence that the work has been copied from the 
public domain and evidence that the work was non-copyrightable. Id. (citing Folio Impressions, 
Inc. v. Byer Cal., 937 F.2d 759, 763-64 (2d Cir. 1991); Carol Barnhart, Inc. v. Economy Cover 
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Corp., 773 F.2d 411, 414 (2d Cir. 1985)). Parties challenging the validity of copyright 
registrations must therefore do more than simply point out potential errors in the certificate. See 
2 Nimmer on Copyright § 7.20(b)(1) (“a misstatement... in the registration application, if 
unaccompanied by fraud, should neither invalidate the copyright nor render the registration 
certificate incapable of supporting an infringement action”). 

The ASTM Plaintiffs produced copyright certificates for each of the nine standards at 
issue, and each of these certificates list the ASTM Plaintiffs as the authors of the works. 4 The 
AERA Plaintiffs also produced the copyright certificates for the 1999 Standards, listing the 
AERA Plaintiffs as authors. 5 Two of ASTM’s standards—D86-07 and D975-07—were 
registered more than five years after they were published. The court accords these the same 
evidentiary weight as if they had been registered within five years. See 17 U.S.C. § 410(c) (court 
has discretion over evidentiary weight). Moreover, the court finds that the registration certificate 
for the 1999 Book of Standards sufficiently establishes prima facie evidence of ASTM’s 
ownership of D396-98 and D1217-93(98). Therefore, the ASTM Plaintiffs and AERA Plaintiffs 
have established their ownership of the works at issue with prima facie evidence. 


4 The nine copyright registrations are provided in the record here: 

■ ASTM : Ex. 1 to O’Brien Decl. (ASTM D86-07) (ASTM ECF No. 118-7, p. 13); Ex. 2 to 
O’Brien Decl. (ASTM D975-07) (ASTM ECF No. 118-7, p. 16); Ex. 4 to O’Brien Decl. 
(1999 Annual Book of ASTM Standards) (ASTM ECF No. 118-7, p. 23); Ex. 3 to O’Brien 
Decl. (listing ASTM D396-98 and ASTM D1217-93(98) as standards included in the 1999 
Annual Book of ASTM Standards) (ASTM ECF No. 118-7, pp. 20-21). 

■ NFPA : Ex. A to Berry Decl. (National Electrical Code, 2011 ed.) (ASTM ECF No. 118-3, 
p. 6); Ex. B to Berry Decl. (2014 ed.) (ASTM ECF No. 118-3, p. 8). 

■ ASHRAE : Ex. 3 to Reiniche Decl. (Standard 90.1, 2004 ed.) (ASTM ECF No. 118-10, 
page 16); Ex. 4 to Reiniche Decl. (2007 ed.) (ASTM ECF No. 118-10, page 19); Ex. 5 to 
Reiniche Decl. (2010 ed.) (ASTM ECF No. 118-10, page 22). 

5 Ex. RRR to Levine Decl. (original copyright registration) (AERA ECF No. 60-83); Ex. SSS to 
Levine Decl. (2014 corrected registration) (AERA ECF No. 60-84). 
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The burden to offer evidence disproving ownership thus shifts in both cases to Defendant. 
See Zanzibar, 698 F. Supp. 2d at 202; Roeslin v. District of Columbia, 921 F. Supp. 793, 797 
(D.D.C. 1995) (finding that because the copyright registration listed plaintiff as the author, the 
“burden is thus on the defendant to establish” that plaintiff was not the author). To rebut the 
presumption of validity, in both cases Defendant pointed to the fact that the certificates state that 
the standards were “works for hire”—i.e., that Plaintiffs acquired authorship and ownership 
rights because their employees or anyone who signed a work-for-hire agreement wrote the 
standards—and the certificates further state that Plaintiffs are the authors of the “entire text[s],” 
when Plaintiffs have said that the standards are drafted by hundreds or thousands of volunteer 
contributors. Defendant contends that the certificates must list all of these hundreds or thousands 
of authors in order to be accurate, and that the failure to do so is a material error which strips 
Plaintiffs of the presumption of ownership. However, Defendant offers scant support for this 
argument. 

Moreover, Defendant failed to meet its initial burden, since it did not adduce any 
additional evidence disproving Plaintiffs’ authorship. Instead, Defendant points to weaknesses 
in the additional evidence that Plaintiffs proffered to establish their ownership, including 
questioning whether every one of the hundreds of Plaintiffs’ members who contributed to the 
standards at issue signed an agreement with appropriate language transferring or assigning 
copyright ownership to Plaintiffs. Because Plaintiffs may have standing to bring this 
infringement suit even as part owners of the copyrights, it is not clear why Defendant asserts that 
Plaintiffs must prove outright ownership of their copyrights. Beyond showing that Plaintiffs’ 
recordkeeping could perhaps be more thorough, Defendant has not identified any evidence that 
either the ASTM Plaintiffs or AERA Plaintiffs do not own the copyrights of the standards, in 
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whole or in part. The court therefore concludes that the ASTM Plaintiffs and AERA Plaintiffs 
are the owners of the copyrights at issue and have standing to bring their claims. 6 
b. Valid Copyrights 

Defendant also argues that Plaintiffs do not own “valid” copyrights under Feist because 
the standards either were never copyrightable or lost their copyright protection upon 
incorporation by reference into federal regulations. Defendant argues that the standards cannot 
be copyrighted because: (1) they are methods or systems, which are not entitled to protection 
under 17 U.S.C. § 102(b); (2) the standards are in the public domain as “the law”; and (3) the 
merger and scenes a faire doctrines preclude a finding of infringement. 

(i). Methods or Systems under Copyright Act § 102(b) 

Section 102(b) of the Copyright Act specifies eight types of works that are not protected 
by copyright: “In no case does copyright protection for an original work of authorship extend to 
any idea, procedure, process, system, method of operation, concept, principle, or discovery, 
regardless of the form in which it is described, explained, illustrated, or embodied in such work.” 

17 U.S.C. § 102(b). Though these eight types of works are not further defined in the statute, the 
legislative history accompanying the Copyright Act of 1976 offers some starting guidance: 

“Section 102(b) in no way enlarges or contracts the scope of copyright protection under the 
present law. Its purpose is to restate, in the context of the new single Federal system of 
copyright, that the basic dichotomy between expression and idea remains unchanged.” H.R. 


6 Defendant did not dispute that “ASTM has copyright registrations that cover each of the 
standards at issue in this litigation” except as to one standard, ASTM D323-58(68). (See Def. 
Statement of Disputed Facts 170 (ASTM ECF No. 121-3)). Therefore, unless Defendant 
presents evidence disproving ownership, the court is likely to conclude, based on these copyright 
registrations, that the ASTM Plaintiffs are the owners of the remaining standards at issue in this 
litigation, with the exception of D323-58(68). As to this standard, ASTM will need to present 
additional evidence establishing ownership. 
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Rep. No. 94-1476, at 57, reprinted in 1976 U.S.C.C.A.N. 5659, 5670 (Sept. 3, 1976); S. Rep. 

No. 94-473 (Nov. 20, 1975); see also 1-2A Nimmer on Copyright § 2A.06(a)(l) (summarizing 
legislative history). The “basic dichotomy” refers to the well-established principle that ideas 
ca nn ot be copyrighted, but expression of those ideas can be. See 1-2A Nimmer on Copyright 
§ 2 A. 06(a)(2)(b) (a work “is to be denied protection only if that protection would be tantamount 
to protecting an excluded category (e.g., idea or method of operation) without regard to the fact 
that the excluded subject matter is expressed or embodied in expression”). 

This section of the Copyright Act codifies the Supreme Court’s 1879 decision in Baker v. 
Selden, 101 U.S. 99 (1897), which denied copyright protection for systems, methods, processes, 
and ideas. Baker evaluated a copyright claim by the author of a manual describing “a peculiar 
system of book-keeping” against a defendant who published a similar guide to book-keeping 
using “a similar plan so far as results are concemed[,] but mak[ing] a different arrangement of 
the columns, and us[ing] different headings.” Id. at 100. The Court defined the question as 
“whether the exclusive property in a system of book-keeping can be claimed, under the law or 
copyright, by means of a book in which that system is explained.” Id. at 101. In answering this 
question, the Court offered as an example that “[t]he copyright of a work on mathematical 
science cannot give to the author an exclusive right to the methods of operation which he 
propounds, or to the diagrams which he employs to explain them, so as to prevent an engineer 
from using them whenever occasion requires.” Id. at 103. This distinction between the actual 
method or system described by a work, which cannot be copyrighted, and the written words 
describing it, which can, is fundamental to understanding the Copyright Act’s modem limitations 
to copyright protection in § 102(b). 

Defendant primarily argues that the Plaintiffs’ standards are completely devoid of 
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creative expression and are merely recitations of processes or procedures that a person or entity 
would follow. Part of this argument appears to rest only on the fact that the names of the ASTM 
Plaintiffs’ standards, and their descriptions or advertisements, include the words “method” and 
“procedure.” See, e.g., ASTM D86-07 Standard Test Method for Distillation of Petroleum 
Products at Atmospheric Pressure, Ex. 6 to Decl. of Thomas O’Brien (“O’Brien Decl.”) (ASTM 
ECF No. 118-7 at 107)); ASTM D1217-93(98) Standard Test Method for Density and Relative 
Density (Specific Gravity) of Liquids by Bingham Pycnometer, Ex. 9 to O’Brien Decl. (ASTM 
ECF No. 118-7 at 136). Additionally, the AERA Plaintiffs’ Rule 30(b)(6) representative noted 
that the 1999 Standards “describe procedures, statistical procedures, research procedures .. . how 
to design a test, how to collect evidence of validity, [and] how to calculate the reliability of 
tests.” (Def. Br. at 32 (citing AERA DSMF ][ 77)). However, simply calling a work a 
“procedure” or a “method” does not revoke its copyright protection under the Copyright Act. 

This argument misunderstands or ignores the expression/idea dichotomy rooted in Baker and 
codified in § 102(b). 

Defendant also emphasizes that because the Plaintiffs’ standards are highly technical, 
complex, and precise, and because testimony shows that the ASTM Plaintiffs attempt to create 
the “best” standards, then the standards are “dictated by utility” or just “discovered facts,” and 
lack any creative expressive content. However, the court rejects the argument that voluntary 
consensus standards, such as those here, are analogous to a list of ingredients or basic 
instructions in a recipe, or a series of yoga poses, as in the cases cited by Defendant. Not only is 
there a vast gulf between the simplicity of an ingredient list and the complexity of the standards, 
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but, more importantly, the standards plainly contain expressive content. 7 As one example, 

ASTM D1217-93 lists under the heading “Significance and Use”: “Although [the standard] is no 
longer employed extensively for the purpose, this test method is useful whenever accurate 
densities of pure hydrocarbons or petroleum fractions with boiling points between 90 and 110°C 
are required.” (ASTMECFNo. 118-7 at 136). 

The standards in these cases contain expression that is certainly technical but that still 
bears markings of creativity. As the Supreme Court instructed in Feist, “the requisite level of 
creativity is extremely low; even a slight amount will suffice. The vast majority of works make 
the grade quite easily, as they possess some creative spark, ‘no matter how crude, humble or 
obvious’ it might be.” 499 U.S. at 345 (quoting 1 M. Nimmer & D. Nimmer, Copyright 
§ 1.08(C)(1) (1990)). Moreover, as Defendant conceded, there are many possible forms of 
expression through which the technical material in the standards could be conveyed, and the 
volunteer and association members who collectively author the standards “debate wording in the 
standards.” (Def. Br. at 32 (ASTM ECF No. 121)). Thus, however “humble” or “obvious” 
Defendant finds the Plaintiffs’ creative choices, the standards still bear at least the “extremely 
low” amount of creativity required by the Supreme Court. Moreover, the undisputed record 
evidence also shows that other parties have written different standards on the same exact subject 
matter as ASTM Plaintiffs’ standards, undermining the argument that the standards are so 
technical and precise there can be only one possible expression. (ASTM PSMF ffl[ 38, 133). 

Importantly, Baker and § 102(b) bar Plaintiffs from attempting to copyright the system or 


7 Defendant does not request that this court scour the over 1,000 pages of the nine of ASTM 
Plaintiffs’ standards provided to the court or the over 200 pages of the 1999 Standards, and the 
court was not provided with copies of the remaining standards. The court declines to engage in 
such an exercise here. 
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method itself, not the written work explaining or describing that method. Here, the copyright 
protections held by the Plaintiffs do not prevent any person or entity from using or applying the 
procedures described in the standards, only from copying their written descriptions of those 
standards. Defendant presented no evidence that the Plaintiffs have sought to block an entity or 
person from using the procedures described in the standards. In fact, use of the procedures 
described is the entire purpose of such voluntary consensus standards. The court therefore 
concludes that § 102(b) of the Copyright Act does not preclude these standards from being 
copyrighted. 

(ii). Loss of Copyright Upon Entering the Public Domain 
A. Federal Law Does Not Bar Copyrightability 
At the heart of Defendant’s defense is the argument that Plaintiffs’ standards lost their 
copyright protections the instant they were incorporated by reference into federal regulations. 

There are weighty policy arguments on both sides of this issue, including the need to preserve a 
vital and complicated public-private partnership between the government and SDOs, and the 
need for an informed citizenry to have a full understanding of how to comply with the nation’s 
legal requirements. However, this suit is not about access to the law in a broad sense, but instead 
about the validity of copyrights for these standards under current federal law. Copyright 
protection is a creature of statute, and as such is the result of careful policy considerations by 
Congress. In the view of this court, Congress has already passed on the question of revoking 
copyright protection for standards that have been incorporated by reference into regulations, and 
any further consideration of the issue must be left to Congress for amendment. 

Section 105 of the Copyright Act states that “[cjopyright protection under this title is not 
available for any work of the United States Government.” 17 U.S.C. § 105. The Act defines a 
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“work of the United States Government” as “a work prepared by an officer or employee of the 

United States Government as part of that person’s official duties.” Id. § 101. These are the only 

government-related works that outright lack copyright under the law. For other types of works, 

such as those commissioned by the government or created under government contract by private 

parties, Congress chose to make case-by-case decisions and leave the determination of whether 

private copyright should exist to the federal agency that commissioned or contracted for the 

work. The House Report accompanying the Copyright Act states: 

The bill deliberately avoids making any sort of outright, unqualified prohibition 
against copyright in works prepared under Government contract or grant. There 
may well be cases where it would be in the public interest to deny copyright in the 
writings generated by Government research contracts and the like; it can be 
assumed that, where a Government agency commissions a work for its own use 
merely as an alternative to having one of its own employees prepare the work, the 
right to secure a private copyright would be withheld. However, there are almost 
certainly many other cases where the denial of copyright protection would be 
unfair or would hamper the production and publication of important works. 

Where, under the particular circumstances, Congress or the agency involved finds 
that the need to have a work freely available outweighs the need of the private 
author to secure copyright, the problem can be dealt with by specific legislation, 
agency regulations, or contractual restrictions. 

H.R. Rep. No. 94-1476, at 5672 (1976), reprinted in 1976 U.S.C.C.A.N. 5659, 5672. 

Defendant argues that Sections 102(b) (no protection for systems or methods) and 105 

(no protection for Government-authored works) should be read together to indicate that Congress 

intended that there be no copyright protections for incorporated standards because, like judicial 

opinions—which the Supreme Court nearly two hundred years ago determined could not be 

copyrighted—the standards, once incorporated, are “legal facts” which cannot be copyrighted. 

See Wheaton v. Peters, 33 U.S. 591, 668 (1834) (writing that the Court was “unanimously of the 

opinion that no reporter has or can have any copyright in the written opinions delivered by this 

Court”); Banks v. Manchester, 128 U.S. 244, 253 (1888) (“The whole work done by the judges 
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constitutes the authentic exposition and interpretation of the law, which, binding every citizen, is 
free for publication to all, whether it is a declaration of unwritten law, or an interpretation of a 
constitution or a statute.”). While these cases form the bedrock for the long-standing principle 
that works authored by government officials or employees cannot be copyrighted, the cases 
involved works by actual government officials—i.e., judges—acting in their official capacity, 
unlike here. That was the principle codified in § 105 of the Copyright Act and restated in the 
U.S. Copyright Office’s Compendium of Copyright Office Practices § 313.6(c)(2) (3d ed. 2014), 
which states: “As a matter of longstanding public policy, the U.S. Copyright Office will not 
register a government edict that has been issued by any state, local, or territorial government, 
including legislative enactments, judicial decisions, administrative rulings, public ordinances, or 
similar types of official legal materials.” 

Congress was well aware of the potential copyright issue posed by materials incorporated 
by reference when it crafted Section 105 in 1976. Ten years earlier, Congress had extended to 
federal agencies the authority to incorporate private works by reference into federal regulations. 

See Pub. L. No. 90-23, § 552, 81 Stat. 54 (1967) (codified at 5 U.S.C. § 552) (providing that 
“matter reasonably available to the class of persons affected thereby is deemed published in the 
Federal Register when incorporated by reference therein with the approval of the Director of the 
Federal Register”). However, in the Copyright Act of 1976, Congress made no mention of these 
incorporated works in § 105 (no copyright for “any work of the United States Government”) or 
any other section. As the House Report quoted above indicates, Congress already carefully 
weighed the competing policy goals of making incorporated works publicly available while also 
preserving the incentives and protections granted by copyright, and it weighed in favor of 
preserving the copyright system. See H.R. Rep. No. 94-1476, at 60 (1976) (stating that under 
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§ 105 “use by the Government of a private work would not affect its copyright protection in any 
way”); see also M.B. Schnapper v. Foley, 667 F.2d 102, 109 (D.C. Cir. 1981) (analyzing 
Copyright Act and holding that “we are reluctant to cabin the discretion of government agencies 
to arrange ownership and publication rights with private contractors absent some reasonable 
showing of a congressional desire to do so”). 

However, recognizing the importance of public access to works incorporated by reference 
into federal regulations, Congress still requires that such works be “reasonably available.” 5 
U.S.C. § 552(a)(1). Under current federal regulations issued by the Office of the Federal 
Register in 1982, a privately authored work may be incorporated by reference into an agency’s 
regulation if it is “reasonably available,” including availability in hard copy at the OFR and/or 
the incorporating agency. 1 C.F.R. § 51.7(a)(3). Thirteen years later, Congress passed the 
National Technology Transfer and Advancement Act of 1995 (“NTTAA”) which directed all 
federal agencies to use privately developed technical voluntary consensus standards. See Pub. L. 
No. 104-113, 110 Stat. 775 (1996). Thus, Congress initially authorized agencies to incorporate 
works by reference, then excluded these incorporated works from § 105 of the Copyright Act, 
and, nearly twenty years later, specifically directed agencies to incorporate private works by 
reference. From 1966 through the present, Congress has remained silent on the question of 
whether privately authored standards and other works would lose copyright protection upon 
incorporation by reference. If Congress intended to revoke the copyrights of such standards 
when it passed the NTTAA, or any time before or since, it surely would have done so expressly. 

See Whitman v. Am. Trucking Ass’ns, Inc., 531 U.S. 457, 468 (2001) (“Congress . . . does not 
alter the fundamental details of a regulatory scheme in vague terms or ancillary provisions—it 
does not. . . hide elephants in mouseholes.”); United States v. Fausto, 484 U.S. 439, 453 (1988) 

22 


JA2080 


Case l:13-cv-01215-TSC Document 175 Filed 02/02/17 Page 23 of 55 
USCA Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 322 of 573 

(“[It] can be strongly presumed that Congress will specifically address language on the statute 
books that it wishes to change.”). Instead, Congress has chosen to maintain the scheme it created 
in 1966: that such standards must simply be made reasonably available. See 5 U.S.C. 

§ 552(a)(1). 

Moreover, Congress has similarly determined that online access to the nation’s laws and 
regulations need not be provided for no cost. In establishing “a system of online access to the 
Congressional Record [and] the Federal Register,” Congress authorized the Superintendent of 
Documents, under the direction of the Director of the Government Publishing Office, to “charge 
reasonable fees for use of the directory and the system of access.” 44 U.S.C. §§ 4101-02. While 
citing this statute and noting that the Superintendent has chosen not to charge fees for online 
access, OFR in its 2013 proposed rulemaking stated that Congress had not made a policy 
determination that online access to the law must be provided free of charge. See Incorporation 
by Reference, 78 Fed. Reg. 60,784, 60,785 (Oct. 2, 2013). Similarly, OFR recently determined 
that “reasonably available” under § 552(a)(1) did not mean availability for no cost on the 
Internet. See id. (considering proposed amendments to OFR’s regulations on incorporation by 
reference and specifically addressing and rejecting the argument that standards incorporated by 
reference should be posted online for free in order to be reasonably available). 

Importantly, there is no evidence that the ASTM Plaintiffs’ standards or the AERA 
Plaintiffs’ standards are unavailable to the public. In fact, the undisputed record evidence shows 
that the standards are required to be available in physical form from OFR (see 1 C.F.R. 

§ 51.3(b)(4)); are available for purchase from the AERA Plaintiffs in hard copy (AERA PSMF 
134) and from the ASTM Plaintiffs in hard copy and PDFs (see ASTM PSMF 157, 99, 157); 
and are accessible in read-only format for free in ASTM Plaintiffs’ online reading rooms (see 
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ASTM PSMF ]| 64, 100, 161). While Defendant argues that the public requires greater access to 
the standards—in particular, free online access in formats other than read-only—that is a policy 
judgment best left to Congress. The arguments raised by the parties and by amici highlight 
important considerations regarding unrestricted access to the texts of laws, regulations, and 
incorporated materials, as well as the strong need to protect the economic incentives for the 
further creation of new standards through revenues from the sale of existing standards. This is 
the policy balancing that Congress is presumed to have already engaged in, and any further 
changes to the law in light of new technological developments and resulting changes in public 
expectations of access to information are best addressed by Congress, rather than this court. 

B. Due Process Concerns Do Not Bar Copyrightability 
Defendant further argues that even if the Copyright Act does not bar copyright protection 
for incorporated standards, individuals have a due process right to access the text of “the law,” 
including the standards at issue here. Four Circuit Courts have considered similar arguments 
regarding copyrighted works incorporated by reference into state and federal regulations. See 
Bldg. Officials & Code Admins, v. Code Tech., Inc., 628 F.2d 730 (1st Cir. 1980) (“BOCA”) 
(declining to rule on the question); CCCInfo. Servs., Inc. v. McLean Hunter Mkt. Reports, Inc., 

44 F.3d 61, 74 (2d Cir. 1994) (upholding copyright in work incorporated by reference); Cnty. of 
Suffolk, N.Y. v. First Am. Real Estate Solutions, 261 F.3d 179 (2d Cir. 2001) (same); Practice 
Mgmt. Info. Corp. v. Reports, Inc., 121 F.3d 516, 518 (9th Cir. 1997) (same); Veeckv. S. Bldg. 

Code Cong. Int’l, Inc., 293 F.3d 791, 796 (5th Cir. 2002) (en banc) (holding that incorporation 
by reference revoked the copyright owner’s copyright protection). The court will briefly 
describe each of these Circuit decisions. 

The question of whether a privately-authored, copyrighted work might lose its copyright 
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protection after being referenced in a law was first discussed by the First Circuit in BOCA. That 
case involved a nonprofit, BOCA, which authored and copyrighted a model code called the 
“Basic Building Code.” See 628 F.3d at 731-32. Massachusetts adopted a building code based 
in substantial part on the BOCA Basic Building Code, called the Co mm onwealth of 
Massachusetts State Building Code. Id. at 732. BOCA sold a printed version of the 
Massachusetts State Building Code for $22 a copy, and the state referred any persons interested 
in obtaining a copy of the code for their own use to BOCA. Id. The defendant, Code Tech., Inc., 
published its own copy of the Massachusetts State Building Code and sold it for $35 per volume. 

Id. In the subsequent copyright infringement suit, the district court granted BOCA’s request for 
a preliminary injunction, and the First Circuit reversed, though it reserved judgment on the 
merits of whether the building code was validly copyrighted. Instead, it noted that “[t]he citizens 
are the authors of the law, and therefore its owners, regardless of who actually drafts the 
provisions, because the law derives its authority from the consent of the public, expressed 
through the democratic process.” Id. at 734. 

The Second Circuit considered similar issues in two cases. First, in CCC, the court 
considered whether copyright protection for a compilation called the Red Book, which listed 
used car valuations, was revoked after it was referenced by states as one of several references for 
car valuation. See 44 F.3d at 74. The court rejected the argument that referenced works enter 
the public domain, stating: “We are not prepared to hold that a state’s reference to a copyrighted 
work as a legal standard for valuation results in loss of the copyright. While there are indeed 
policy considerations that support [defendant’s public domain] argument, they are opposed by 
countervailing considerations.” Id. The court then analogized to a state education system 
assigning copyrighted books as a mandatory part of a school curriculum and noted that under the 
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public domain logic, these books might lose copyright protection. Id. 

Second, in County of Suffolk, the Second Circuit considered the copyrightability of a 
county’s tax maps. The court looked to Banks, in which the Supreme Court held that judicial 
opinions were not copyrightable, and determined that Banks established two premises: (1) that 
judges’ opinions cannot be copyrighted because judges receive their salaries from the public 
treasury and do not have the economic incentives that copyrights are designed to protect; and 
(2) there are due process considerations because the “whole work done by the judges constitutes 
the authentic exposition and interpretation of the law, which, binding every citizen, is free for 
publication to all.” 261 F.3d at 193-94 (citing Banks v. Manchester, 128 U.S. 244, 253 (1888)). 
Building on these premises, the Second Circuit articulated two factors that should guide courts’ 
analysis in these situations: first, “whether the entity or individual who created the work needs 
an economic incentive to create or has a proprietary interest in creating the work”; and second, 
“whether the public needs notice of this particular work to have notice of the law.” Id. at 194 
(citing Practice Management, 121 F.3d at 518-19; BOCA, 628 F.2d at 734-35). With regard to 
this second factor, the court primarily considered the severity of criminal or civil sanctions 
associated with failure to adhere to the maps at issue. Finding no serious penalties, it focused on 
the fact that citizens had “fair warning” of the tax maps from their reference in the tax statute, 
and there was “no allegation that any individual required to pay the applicable property tax ha[d] 
any difficulty in obtaining access to either the law or the relevant tax map.” Id. at 195. 

Therefore, the maps were entitled to copyright protection. 

Like the Second Circuit, the Ninth Circuit in Practice Management also decided to 
preserve the copyright protections in the American Medical Association’s (“AMA”) publication 
of medical codes and descriptions which had been incorporated by reference by the U.S. Health 
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Care Financing Administration (“HCFA”). Under the HCFA’s regulation, parties seeking health 

insurance reimbursement for Medicare were required to use the codes created and copyrighted 

by the AMA. See 121 F.3d at 518. The Ninth Circuit similarly looked to Banks and focused on 

its premise that there is a due process interest in free access to the law. Like the Second Circuit, 

the court considered this due process interest and ultimately rejected revoking the AMA’s 

copyright because “[tjhere [was] no evidence that anyone wishing to use the [copyrighted codes] 

ha[d] any difficulty obtaining access to it.” Id. at 519. 

Finally, counter to the opinions of other circuits, the Fifth Circuit sitting en banc in Veeck 

focused more heavily on the first Banks premise regarding economic incentives and held that 

copyright protection is revoked when a model code is adopted as law by a municipality, stating 

that “as law, the model codes enter the public domain and are not subject to the copyright 

holder’s exclusive prerogatives.” 293 F.3d at 793. However, the court carefully distinguished its 

decision from the facts in the aforementioned cases. It wrote: 

[T]he limits of this holding must be explained. Several national standards-writing 
organizations joined [defendant] as amici out of fear that their copyrights may be 
vitiated simply by the common practice of governmental entities’ incorporating 
their standards in laws and regulations. This case does not involve references to 
extrinsic standards. Instead, it concerns the wholesale adoption of a model code 
promoted by its author, [defendant], precisely for use as legislation. Caselaw that 
derives from official incorporation of extrinsic standards is distinguishable in 
reasoning and result.... If a statute refers to the Red Book or to specific school 
books, the law requires citizens to consult or use a copyrighted work in the 
process of fulfilling their obligations. The copyrighted works do not ‘become 
law’ merely because a statute refers to them. . . . Equally important, the 
referenced works or standards in CCC and Practice Management were created by 
private groups for reasons other than incorporation into law. To the extent 
incentives are relevant to the existence of copyright protection, the authors in 
these cases deserve incentives.... In the case of a model code, on the other hand, 
the text of the model serves no other purpose than to become law. 

Id. at 803-05. The cases before the court, involving some of the same amici referenced in Veeck, 

do not involve model codes adopted verbatim in their entirety into legislation. Instead, the 
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standards incorporated by reference provide guidelines and procedures that individuals or entities 
must use or reference in the fulfillment of their legal obligations under federal regulations. 

Applying the first premise of Banks to the facts here, Defendant argues that Plaintiffs do 
not require economic incentives to create their standards because they actively lobby and 
advocate for their standards to be incorporated by reference into regulations, including investing 
funds on lobbying to that effect. Therefore, Defendant argues, the court should find that 
Plaintiffs create standards for no purpose other than adoption into law, as the Veeck court 
determined regarding the model code in that case. Here however, the facts indicate that 
Plaintiffs create standards for a wide range of industries, that the majority of their standards are 
not incorporated into regulations, and that even those that have been incorporated by reference 
have undergone updates and revisions to reflect modern use, despite the regulations 
incorporating past versions. Plaintiffs and supporting amici highlight that without copyright 
protection for all of their standards, they will face significant difficulty raising the necessary 
revenue to continue producing high-quality voluntary consensus standards. In its Notice of 
Proposed Rulemaking, OFR relied on this same argument to ultimately reject a proposal to 
require free online access to standards in its “reasonably available” determination. 78 Fed. Reg. 
at 60,785 (“If we required that all materials IBR’d into the CFR be available for free, that 
requirement would compromise the ability of regulators to rely on voluntary consensus 
standards, possibly requiring them to create their own standards, which is contrary to the 
NTTAA and the OMB Circular A-l 19.”). 

As for the second premise of Banks, this court finds that, as in the cases before the 
Second and Ninth Circuits, there is no evidence here that anyone has been denied access to the 
standards by the ASTM Plaintiffs or AERA Plaintiffs. Instead, Defendant simply argues that the 
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public should be granted more expansive access. 

Therefore, considering the Banks holdings and given the existing statutory, regulatory, 
and judicial framework, this court finds that Plaintiffs’ standards have not entered the public 
domain upon their incorporation by reference into federal regulations and do not lose their 
copyright protection. This conclusion does not dismiss or diminish the valid public policy 
concern that citizens benefit from greater access to statutes, regulations, and all materials they 
must reference in fulfilling their legal obligations. The ability to know, understand, and 
communicate the law as a broad concept is of paramount importance to the continued success of 
our democracy. However, changes to the statutory or regulatory framework that reconsider the 
balancing of interests underlying modem copyright law and incorporation by reference must be 
made by Congress, not this court. 

(Hi). Merger Doctrine 

Defendant asks the court to apply the “merger doctrine” to find that the standards cannot 
be copyrighted because the expressions in the standards have merged with the law to become 
facts. Under modem copyright law, there is a well-known dichotomy between “expression,” 
which can generally be copyrighted, and “ideas,” which cannot. 4-13 Nimmer on Copyright 
§ 13.03. The merger doctrine has developed to consider those specific situations in which “the 
idea ‘merges’ with the expression, such that a given idea is inseparably tied to a particular 
expression.” Id. at § 13.03(3). This can occur when there “are so few ways of expressing an 
idea [that] not even the expression is protected by copyright.” Id. (quoting BUC Int’l Corp. v. 

Int’l Yacht Council Ltd., 489 F.3d 1129, 1143 (11th Cir. 2007)). 

The parties disagree as to the proper merger doctrine analysis. Defendant argues that 
upon their incorporation by reference, the standards become “merged” with the “fact” that is the 
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law. Plaintiffs argue that to determine if an idea and expression have merged, the court should 
focus on whether there were any other ways of articulating a particular idea when the work was 
first published, not when it was later incorporated by reference. In essence, the parties disagree 
as to whether the merger doctrine is a question of copyrightability—meaning the Plaintiffs’ 
standards might lose copyright protection upon incorporation by reference—or an affirmative 
defense to copyright infringement—i.e., the allegedly infringing work did not violate copyright 
because there was no other way to express the content of the work. Plaintiffs argue that the 
merger doctrine addresses only the question of copyrightability, and so the court’s analysis 
should focus on whether, at the time the standards were authored, there were no other ways to 
articulate and arrange such standards. Defendant contends that the standards could not be 
expressed any other way after incorporation into regulations, and thus its display of the standards 
was not infringement. 

The court declines to resolve this merger doctrine issue, since under either approach, the 
standards maintain copyright protection. At the time they were authored, there were certainly 
myriad ways to write and organize the text of the standards, and, for the reasons discussed above, 
the standards did not lose their copyright protections upon incorporation by reference into federal 
regulations. Therefore, the merger doctrine neither precludes a finding of copyrightability nor 
serves as a defense for Defendant. 

(iv). Scenes a Faire Doctrine 

Finally, Defendant points to the scenes a faire doctrine, which similarly may be 
approached as a question of copyrightability or an affirmative defense. The doctrine typically 
applies to “incidents, characters, or settings which are as a practical matter indispensable, or at 
least standard, in the treatment of a given topic.” Nimmer § 13.03(4) (quoting Atari, Inc. v. 
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North Am. Philips Consumer Elecs. Corp., 672 F.2d 607, 616 (7th Cir. 1982), cert, denied, 459 
U.S. 880 (1982)). Ni mm er offers examples such as the use of a bar room scene in a film about a 
broken-hearted lover because, as the name of the doctrine suggests, these are “scenes which must 
be done.” Id. Defendant argues here that Plaintiffs’ standards are entirely “uncopyrightable” 
because they are “shaped by external factors,” such as the desire to satisfy regulations and laws 
and to write what Plaintiffs believe to be the most accurate and clear standards. (Tr. of Motions 
Hearing at 62:15-19 (ASTM ECF No. 173); Def. Br. at 34). However, this doctrine is a poor fit 
for Defendant’s arguments. In the court’s view, there is a great deal of difference between every 
detail of the phrasing, explanation, and organization across thousands of pages of standards, 
which Defendant argues is entirely dictated by Plaintiffs’ broad desires for accuracy and clarity, 
and the inclusion of a generic bar room scene in a romantic drama where the audience expects it. 
Defendant offers no cases to support its argument that this doctrine bars copyrightability of the 
standards at issue here, and this court knows of none. The court concludes that the scenes a faire 
doctrine does not act as a bar to the copyrightability of Plaintiffs’ standards and does not serve as 
a defense for Defendant’s display of the standards 

In sum, the court concludes that Plaintiffs own valid copyrights over the standards at 
issue, and that the copyrights were not stripped upon the incorporation by reference into federal 
regulations. 

2. Feist Prong 2: Copying an Original Work 

a. Overview 

Having established that both the ASTM Plaintiffs and AERA Plaintiffs own valid 
copyrights in the standards at issue, the second question for the court under Feist is whether 
Public Resource, by scanning and posting online the standards at issue “cop[ied] anything that 
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was ‘original’ to” the Plaintiffs. Feist, 499 U.S. at 361. Copying means exercising any of the 
exclusive rights that 17 U.S.C. § 106 vests in the owners of a copyright. See Call of the Wild 
Movie, LLC v. Does, 770 F. Supp. 2d 332, 351 (D.D.C. 2011). These rights include the rights of 
reproduction, distribution, display, and creation of derivative works. See 17 U.S.C. § 106(1)—(3), 
(5). There is no factual dispute that Public Resource reproduced and posted online for display or 
distribution the standards at issue in this case. Having rejected the application of the merger 
doctrine or scenes a faire doctrine as affirmative defenses, Defendant’s only argument on this 
second prong is therefore that its copying and posting of the standards was “fair use.” 
b. Affirmative Defense of Fair Use 

Under the Copyright Act, fair use of a copyrighted work “is not an infringement of 
copyright.” 17 U.S.C. § 107. Fair use is a defense to a claim of copyright infringement in order 
to “fulfill copyright’s very purpose, ‘to promote the Progress of Science and useful Arts.’” 
Campbell v. Acuff-Rose Music, Inc., 510 U.S. 569, 575 (1994) (quoting U.S. Const, art. I, § 8, cl. 
8). The Copyright Act provides that: 

In determining whether the use made of a work in any particular case is a fair use, 
the factors to be considered shall include— 

(1) the purpose and character of the use, including whether such use is of a 
commercial nature or is for nonprofit educational purposes; 

(2) the nature of the copyrighted work; 

(3) the amount and substantiality of the portion used in relation to the 
copyrighted work as a whole; and 

(4) the effect of the use upon the potential market for or value of the 
copyrighted work. 

17 U.S.C. § 107. The statute further lists examples of uses that are “fair use,” including 
“criticism, comment, news reporting, teaching (including multiple copies for classroom use), 
scholarship, or research.” Id. The fair use doctrine calls for a “case-by-case analysis,” and the 
four statutory factors are meant to provide “general guidance,” weighed together “in light of the 
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purposes of copyright.” Campbell, 510 U.S. at 578-79. 

(i). Purpose and Character of Defendant’s Use of the Standards 

With regard to the first factor, the statute itself offers guidance on the types of purposes 
that might be considered fair use: criticism, commentary, news reporting, teaching, or research. 

Id. § 107. Moreover, the Supreme Court has held that courts should focus on whether the new 
work “supersede^] the objects of the creation ... or instead adds something new, with a further 
purpose or different character, altering the first with new expression, meaning, or message; [the 
question], in other words, [is] whether and to what extent the new work is transformative.” 
Campbell, 510 U.S. at 578-79 (internal quotations omitted). Given the constitutional goal of 
copyright—to promote the development of science and the arts—“the more transformative the 
new work, the less will be the significance of other factors, like commercialism, that may weigh 
against a finding of fair use.” Id. at 579. 

It is undisputed that Public Resource scanned the ASTM Plaintiffs’ standards at issue 
from their physical hardcopies and converted them to searchable PDFs using OCR processing 
(ASTM Pis. SUMF ^ 182) and reproduced some of the standards by re-typing them into HTML 
format. (ASTM PSMF1182; ASTM DSMF | 83). Public Resource scanned the AERA 
Plaintiffs’ 1999 Standards from the physical hard copy and converted them to a PDF file, which 
it then uploaded to its website for display and distribution. (AERA PSMF 69, 71-73; AERA 
DSMF ![ 28). Defendant argues this is transformative in three ways: by providing free access to 
“the law”; by enabling others to use software to analyze the standards; and by enabling those 
with visual impairments to use text-to-speech software. The evidence does not support any of 
these arguments. 

Defendant first argues that it has transformed Plaintiffs’ standards by making identical 
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copies of them and distributing them online for no cost. In Defendant’s view, this is 
transformative because it provides individuals with greater access to “the law.” While Defendant 
argues that its conduct is analogous to those who make copies of copyrighted works in order to 
comply with legal requirements, Defendant was not actually acting to comply with a particular 
law—unlike, for example, an individual who makes a photocopy of the standards located at OFR 
for use on her building project. Instead, Defendant has placed identical copies of Plaintiffs’ 
standards into the online marketplace with no intention to use them itself, but instead to simply 
offer them for free in competition with Plaintiffs’ standards. While Defendant did not earn 
revenue directly from the display of the standards, its activity still bears “commercial” elements 
given that it actively engaged in distributing identical standards online in the same consumer 
market. While this commerciality is not by itself dispositive, it does weigh firmly against fair 
use. See Campbell, 510 U.S. at 594. 

Defendant points to Swatch Group Management Services Ltd. v. Bloomberg L.P., 756 
F.3d 73, 81 (2d Cir. 2014) in support of its proposition that when a copyrighted document is of 
great public importance then posting it online may be transformative. However, Swatch Group 
involved the recording of a private conference call about the company’s earnings report 
involving executives and 132 analysts that Bloomberg then distributed to subscribers of its 
Bloomberg Professional service. Id. at 78-79. Given that Swatch Group instructed call 
participants not to record or broadcast the call, any direct knowledge of what the executives said 
would be limited to those analysts who participated. Id. The facts of Swatch Group do not align 
with those here, where the evidence demonstrates that Plaintiffs’ standards are available to 
anyone for viewing online in ASTM Plaintiffs’ reading rooms, at a public library, at the OFR or 
incorporating agency, or for purchase on Plaintiffs’ websites. This court is unwilling to apply 
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any principles from Swatch Group or similar cases to this case, in which the standards are widely 
available. 

Next, Public Resource argues that distributing the duplicate copies online is 
transformative because, with regard to the ASTM Plaintiffs’ standards, Public Resource first 
altered their formatting through application of OCR or conversion to HTML, which enables 
software analysis or the use of text-to-speech software, and for AERA Plaintiffs’ standards, it 
scanned the hard copy and distributed a PDF version. The court has little difficulty concluding 
that these actions are not transformative. See 4-13 Nimmer on Copyright § 13.05(l)(b); Nihon 
Keizai Shimbun, Inc. v. Comline Bus. Data, Inc., 166 F.3d 65, 72 (2d Cir. 1999) (holding that a 
translation is not a transformative, expressive work); Soc ’y of the Transfiguration Monastery, 

Inc. v. Gregory, 685 F. Supp. 2d 217, 227 (D. Mass. 2010), affirmed, 689 F.3d 29, 59-65 (1st 
Cir. 2012) (“A simple repackaging of a work in a new format, whether on the Internet or on a 
CD-ROM or on a flash drive, is not transformative when the result is simply a mirror image 
reflected on a new mirror.”); see also Authors Guild v. Google, Inc., 804 F.3d 202, 207, 217 (2d 
Cir. 2015) (reasoning Google’s scanning and posting of snippets of copyrighted books online 
was fair use because it made “available information about Plaintiffs’ books without providing 
the public with a substantial substitute for matter protected by the Plaintiffs’ copyright interests 
in the original works or derivatives of them” and added “important value to the basic 
transformative search function, which tells only whether and how often the searched term 
appears in the book”) (emphasis added); Authors Guild, Inc. v. HathiTrust, 755 F.3d 87, 90 (2d 
Cir. 2014) (text searching modification was transformative but where full work was not 
displayed). 

Here, Defendant does not actually perform any analysis on the standards, nor does it offer 
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the service of providing them in an accessible way to those visual impairments. Instead, 

Defendant has identified a series of events that must occur, involving intervening third parties 
and the use of one or more additional software programs, in order for there to be a potentially 
“transformative” use for individuals who are blind or have visual impairments. Defendant in 
both cases proffered the expert report of James Fruchterman, who opined on accessibility of 
written materials for those who are blind. In Fruchterman’s AERA report, he wrote that to make 
a hard copy accessible for those with visual impairments, he would scan the pages, process them 
with OCR to convert the read-only images to searchable text, create a Microsoft Word file, and 
then have it proofread because OCR can create numerous errors. (Expert Rep. of James R. 
Fruchterman at 8 (AERA ECF No. 70-50)). Once such a version is then uploaded online, an 
individual who is blind or visually impaired would then need to use additional screen reader 
software, which “is a program that runs on a personal computer or a smartphone that reads the 
information on the screen aloud (using a computer-synthesized voice) to a blind person.” (Id. at 
3—4). While “most blind people themselves do not have the ability to convert books[,] [s]ome 
blind people have their own home scanners, and if they purchased a used copy online, would be 
able to scan the 1999 Standards page by page on a home scanner, which would take at least two 
hours of labor, and then perform optical character recognition on the title.” (Id. at 8). In his 
ASTM report, Fruchterman wrote that he was able to use a screen reader program to read the text 
of the ASTM Plaintiffs’ standards aloud on Defendant’s website, but not in ASTM Plaintiffs’ 
reading rooms. (Ex. 96 to Becker Deck, Expert Rep. of James R. Fruchterman at 5-7 (ECF No. 
122-6)). Fruchterman noted that some of the PDFs on Defendant’s website were read-only 
images, such as those on ASTM Plaintiffs’ reading rooms, which had to be copied and pasted 
into a Microsoft Word document in order for a screen reader program to operate. (Id. at 16-17). 
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He also noted that individuals who are blind may “independently perform optical character 
recognition on image-based PDFs themselves and access the text that way, and many advanced 
computer users that are blind would be aware that this is possible.” (Id. at 17). He did not opine 
on whether OCR could be performed on the PDFs of standards that ASTM Plaintiffs sell or 
whether he attempted to investigate that as part of his research. 

While it appears Defendant may enable blind individuals, like all other individuals, to 
access the standards at no cost, they still may have to take additional steps like OCR processing 
or converting to a different file type, as well as using additional screen reader programs in order 
to access the standards. There is no evidence that this would not be possible with Plaintiffs’ 

PDFs or by scanning Plaintiffs’ hard copy standards. In Defendant’s view, taking the first step 
or two towards making the standards entirely accessible to those with visual impairments is 
enough to have transformed the standards. This attempts to stretch logic, and certainly the 
doctrine of fair use, too far. Defendant has not offered a sufficiently new purpose to render the 
use transformative, and this weighs against a finding of fair use. 

(ii). Nature of the Copyrighted Standards 

The Supreme Court in Campbell in s tructs that courts should analyze the nature of the 
copyrighted work with “recognition that some works are closer to the core of intended copyright 
protection than others, with the consequence that fair use is more difficult to establish when the 
former works are copied.” 510 U.S. at 586. Many cases create a spectrum between creative, 
fictional expression and factual expression, with the former being “more” protected. See 4-13 
Nimmer § 13.05(A)(2). Defendant argues that Plaintiffs’ standards are “factual,” both because 
they are highly technical and because they are “the law.” However, the Constitution explicitly 
states that copyright exists to “advance the progress of science and the useful arts.” U.S. Const. 
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art. I, § 8, cl. 8. That Plaintiffs’ works involve technical scientific concepts and guidelines does 
not push it away from the core of intended copyright protection, but actually brings it closer. 
Plaintiffs’ standards are vital to the advancement of scientific progress in the U.S. and exactly 
the type of expressive work that warrants full protection under the Constitution and the 
Copyright Act. 

(Hi). Amount and Substantiality of the Portions Defendant Used 
The third factor, “the amount and substantiality of the portion used in relation to the 
copyrighted work as a whole,” 17 U.S.C. § 107(3), weighs overwhelmingly in Plaintiffs’ favor 
and against a finding of fair use. It is undisputed that Defendant copied and distributed identical 
versions of the Plaintiffs’ standards in their entirety. To support its actions as fair use under this 
third factor, Public Resource argues that it was necessary to do so because the full text of the 
standards were incorporated into “the law.” However true it may be that individuals wishing to 
read the text of standards incorporated by reference would want to read them in their entirety, 
this argument is unpersuasive in the fair use analysis. Any market competitor wishing to copy a 
rival’s work and distribute it itself could argue that it “needs” to copy the entire work, otherwise 
its distribution would be less successful. Unsurprisingly, Defendant cannot point to a single case 
that supports its view, and the court finds that this factor also weighs strongly against a finding of 
fair use. 

(iv). Effect of Defendant’s Use Upon Potential Market or Value 
The fourth factor, “the effect of the use upon the potential market for or value of the 
copyrighted work,” 17 U.S.C. § 107(4), “poses the issue of whether unrestricted and widespread 
conduct of the sort engaged in by the defendant would .. . result in a substantially adverse 
impact on the potential market for, or value of, the plaintiffs present work,” 4-13 Nimmer on 
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Copyright § 13.05(A)(4); Campbell, 510 U.S. at 589 (quoting Nimmer). Moreover, the analysis 
“must take into account not only of harm to the original but also of harm to the market for 
derivative works.” Campbell, 510 U.S. at 589 (quoting Harper & Row Publishers, Inc. v. Nation 
Enters., 471 U.S. 539, 568 (1985)). When Defendant engages in “mere duplication for 
commercial purposes,” as here, a harm to the potential market for the copyrighted works may be 
inferred. See id. at 590-91. Such an inference is intuitive based on the facts here where 
consumers in the online marketplace are currently presented with the option to purchase a PDF 
or hard copy version of Plaintiffs’ standards directly from them, or may download a PDF of an 
identical standard for no cost. The only logical conclusion is that this choice negatively impacts 
the potential market for Plaintiffs’ standards. 

In Campbell, the Supreme Court noted that “[s]ince fair use is an affirmative defense, its 
proponent would have difficulty carrying the burden of demonstrating fair use without favorable 
evidence about relevant markets.” 510 U.S. at 590. Here, Defendant did not offer expert 
evidence on the economic impact on the markets, instead pointing to testimony by Plaintiffs’ 
executives that they did not track or know of negative impacts thus far on their revenue from 
Defendant’s conduct. This is not enough to overcome the logical presumption that such activity, 
particularly if it became more widespread by others in the marketplace, would impact Plaintiffs’ 
revenues. It is not Plaintiffs’ burden to establish that they have been harmed in the market, but 
Defendant’s burden to affirmatively establish that such conduct could not even “potentially” 
harm the Plaintiffs’ market. Defendant has not done so. 

(v). Overall Assessment 

Whatever merit there may be in Defendant’s goal of furthering access to documents 
incorporated into regulations, there is nothing in the Copyright Act or in court precedent to 
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suggest that distribution of identical copies of copyrighted works for the direct purpose of 
undermining Plaintiffs’ ability to raise revenue can ever be a fair use. The court thus concludes 
that the fair use doctrine does not serve as a valid defense for Defendant’s conduct. 

Therefore, the court finds that the ASTM Plaintiffs’ motion for summary judgment as to 
their copyright infringement claim is GRANTED, and the AERA Plaintiffs’ motion for summary 
judgment as to their copyright infringement claim is also GRANTED. Defendant’s cross¬ 
motions on copyright infringement are both DENIED. 

B. Contributory Copyright Infringement 

AERA Plaintiffs additionally move for summary judgment on their contributory 
copyright infringement claim. 8 Establishing proof of contributory infringement requires a party 
to demonstrate that the actor was “intentionally inducing or encouraging direct infringement.” 
MGMStudios, Inc. v. Grokster, Ltd., 545 U.S. 913, 930 (2005). Plaintiffs 9 must show (1) direct 
infringement by third parties; (2) that Defendant knew that third parties were directly infringing; 
and (3) that Defendant substantially participated in that direct infringement. Rundquist v. 

Vapiano SE, 798 F. Supp. 2d 102, 126 (D.D.C. 2011). “Merely supplying the means to 
accomplish an infringing activity cannot give rise to the imposition of liability for contributory 
copyright infringement.” Newborn v. Yahoo!, Inc., 391 F. Supp. 2d 181, 186 (D.D.C. 2005) 
(internal quotation omitted). 


8 The ASTM Plaintiffs initially brought a separate claim for contributory copyright 
infringement, but did not include that claim in their motion for summary judgment. Counsel for 
ASTM Plaintiffs stated at oral argument that they believed the remedy for their infringement 
claim covered any potential remedy for their contributory copyright claim. (Tr. of Motions 
Hearing at 122:1-7). 

9 Because ASTM Plaintiffs did not move for summary judgment on their contributory copyright 
claim, for this section the court will use “Plaintiffs” to refer to AERA Plaintiffs. 
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To establish direct infringement by third parties, Plaintiffs must demonstrate “(1) which 
specific original works form the subject of the copyright claim; (2) that the plaintiff owns the 
copyrights in those works; (3) that the copyrights have been registered in accordance with the 
statute; and (4) by what acts [and] during what time the defendant infringed the copyright.” Id. 
(quoting Home & Nature, Inc. v. Sherman Specialty Co., 322 F. Supp. 2d 260, 266 (E.D.N.Y. 
2004)). As discussed above in section III(A), these first three elements have been satisfied. On 
the fourth element, Plaintiffs must show that a third party infringed its copyrights by violating 
their exclusive rights under 17 U.S.C. § 106, including reproduction, preparation of derivative 
works, distribution, or public display. See Home & Nature, 322 F. Supp. 2d at 267. However, 
Plaintiffs only present evidence that the 1999 Standards were “accessed at least 4,164 times” on 
Public Resource’s website and that they were “accessed on the Internet Archive ... website 
1,290 times.” (AERA PSMF 85-86). Without more, there is no basis for the court to 
determine that accessing a website is equivalent to copying or violating any of the exclusive 
rights under § 106. Plaintiffs also assert that “some” individuals “obtained” the standards, but 
their only evidence of this is a redacted e-mail in which an individual states “[0]ne of my 
students showed up for class this semester and told me that he/she didn’t purchase a copy of the 
Standards (I require them as a text for one of my courses) because ‘they are available for free on 
line’ and they showed me the following site.” (Exl. LLL to Deck of Lauress Wise (AERA ECF 
No. 60-75)). Even if such a statement were ultimately determined to be admissible for the truth 
of the matter that the student did not purchase the Standards, it still does not establish that the 
student downloaded or otherwise copied the 1999 Standards from Defendant’s website. 10 


10 The court recognizes that acquiring evidence of downloads may be difficult. Carl Malamud, 
Public Resource’s CEO, testified at deposition that “I don’t know about downloads. It’s 
technically impossible to determine that.” (Ex. A to Hudis Decl. at 347:6-8 (AERA ECF No. 

41 


JA2099 



Case l:13-cv-01215-TSC Document 175 Filed 02/02/17 Page 42 of 55 
USCA Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 341 of 573 

In their Reply Brief, Plaintiffs also point to the possibility that simply browsing a website 
causes a copy of the material on the website to be automatically copied to the computer’s random 
access memory or RAM. See CoStar Realty Info., Inc. v. Field, 737 F. Supp. 2d 496, 507 (D. 

Md. 2010) (analyzing copyright claim involving cache copies of websites in computer’s RAM); 
Ticketmaster, LLCv. RMG Techs., Inc., 507 F. Supp. 2d 1096, 1104-05 (C.D. Cal. 2007) (same). 
While this may be correct, the fact remains that Plaintiffs have put forth no actual evidence that 
even one of the 4,164 accesses resulted in such a copying to a computer’s RAM, and without 
such evidence, Plaintiffs cannot meet their burden on their contributory copyright claim at the 
summary judgment stage. 

The second two factors require Plaintiffs to establish that Defendant knew that third 
parties were engaged in direct infringement and that it substantially participated in such 
infringement. Plaintiffs may demonstrate knowledge by showing that Defendant was notified of 
the third party direct infringement or that it “willfully blind[ed] itself to such infringing uses.” 
Newborn, 391 F. Supp. 2d at 186. On this factor, Plaintiffs again fall short, relying on the fact 
that they asked Defendant to remove the 1999 Standards from its website and Defendant refused 
to do so, as well as evidence that Defendant did not track or prevent downloads of the 1999 
Standards from its website. Without more, this is insufficient to establish that Defendant knew 
that third parties were infringing the Plaintiffs’ copyrights. 

Similarly, Plaintiffs have not presented sufficient evidence on the substantial 
participation factor. While it is undisputed that Defendant posted the 1999 Standards on its 
website to enable greater access for those wishing to read them, because Plaintiffs have not 


60-4)). However, this does not relieve Plaintiffs of the burden of establishing some evidence 
demonstrating direct infringement by third parties. 
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established any actual third party direct infringement, there is insufficient evidence that 
Defendant substantially participated in that infringement. 

Therefore, the court DENIES Plaintiffs’ motion for summary judgment as to its 
contributory copyright claim, and also DENIES Defendant’s motion for summary judgment on 
this claim, as there exists questions of fact as to any third party infringement, Defendant’s 
knowledge, and Defendant’s participation. 

C. Trademark Infringement Claims 

ASTM Plaintiffs additionally moved for summary judgment on their trademark 
infringement, unfair competition and false designation of origin, and common law trademark 
infringement claims, and Defendant cross-moved for summary judgment on these claims as 
well. 11 Trademark law is governed by the Lanham Act, 15 U.S.C. § 1051 etseq., which provides 
that: 

(1) Any person who shah, without the consent of the registrant... (a) use in 
commerce any reproduction, counterfeit, copy, or colorable imitation of a 
registered mark in connection with the sale, offering for sale, distribution, or 
advertising of any goods or services on or in connection with which such use is 
likely to cause confusion, or to cause mistake, or to deceive . . . shah be liable in a 
civil action by the registrant for the remedies hereinafter provided. 

15 U.S.C. §1114(1). In order to prevail on a trademark infringement claim under the Lanham 

Act, Plaintiffs 12 “must show (1) that [they] own[] a valid trademark, (2) that [their] trademark is 

distinctive or has acquired a secondary meaning, and (3) that there is a substantial likelihood of 

confusion between the plaintiffs’] mark and the alleged infringer’s mark.” Globalaw Ltd. v. 

Carmon & Carmon Law Office, 452 F. Supp. 2d 1, 26 (D.D.C. 2006); AARP v. Sycle, 991 F. 


11 The AERA Plaintiffs did not bring a trademark claim, and so this section applies only to 
ASTM Plaintiffs. 

12 As in the preceding section, because only ASTM Plaintiffs moved for summary judgment on 
this claim, the court will refer to them here as Plaintiffs. 
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Supp. 2d 224, 229 (D.D.C. 2013) (same). Common law claims are analyzed under the same 
standard. See AARP, 991 F. Supp. 2d at 229 (citing Breaking the Chain Found., Inc. v. Capitol 
Educ. Support, Inc., 589 F.Supp.2d 25, 29 (D.D.C. 2008)). In order for conduct to be considered 
infringing, there must be a “use in commerce.” 15 U.S.C. §§1114(1), 1125(a)(1). 

Defendant cites Dastar Corp. v. Twentieth Century Fox Film Corp., to discourage the 
court from considering Plaintiffs’ trademark claims on the principle that courts should not 
“misuse or over-exten[d] [] trademark and related protections into areas traditionally occupied by 
patent or copyright.” 539 U.S. 23, 34 (2003). Dastar held that a plaintiff could not bring a false 
designation of origin trademark claim against a defendant who was distributing content that had 
become part of the public domain because the Lanham Act only offers protection “to the 
producer of the tangible goods that are offered for sale, and not to the author of any idea, 
concept, or communication embodied in those goods.” Id. at 37. Unlike in Dastar, Plaintiffs 
here have an independent basis for claiming that Defendant infringed their trademarks, separate 
from their copyright infringement claims: Defendant distributed standards online bearing 
Plaintiffs’ registered trademarks and logos, and Plaintiffs argue that this unauthorized use of their 
marks will confuse consumers and falsely signal that Plaintiffs are the origin of the standards 
distributed on Defendant’s website rather than Defendant. While the remedy sought for 
Plaintiffs’ copyright claim—an injunction barring Defendant from displaying Plaintiffs’ 
standards online—may be broad enough to subsume a remedy for their trademark claims, the 
claims are based on independent arguments, and are therefore the type that Dastar found to be 
appropriate for consideration under the Lanham Act. 

The court must therefore consider whether Plaintiffs own a valid, protectable trademark, 
whether Defendant engaged in an unauthorized use in commerce, whether there is a likelihood of 
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consumer confusion, and whether Defendant’s fair use defense permits its use of the trademarks. 

1. Valid, Protectable Trademark 

Under the Lanham Act, any registration of a trademark “shall be prima facie evidence of 
the validity of the registered mark and of the registration of the mark, of the owner’s ownership 
of the mark, and of the owner’s exclusive right to use the registered mark in commerce.” 15 
U.S.C. § 1057(b). The record indicates that Plaintiffs own valid trademarks of the trademarks 
asserted in this case, and they have federal trademark registrations for each of the asserted 
marks. 13 Thus, Plaintiffs have established a prima facie showing of ownership. Defendant 
offers no evidence to demonstrate that Plaintiffs do not own the trademarks, and therefore the 
court concludes that Plaintiffs are the owners of these marks. 

The trademarks must also be “valid.” To establish validity, Plaintiffs must prove that the 
designation is inherently distinctive or that it has become distinctive by acquiring secondary 
meaning. See Two Pesos, Inc. v. Taco Cabana, Inc., 505 U.S. 763, 769 (1992); Globalaw, 452 
F. Supp. 2d at 26. However, Plaintiffs’ trademark registrations create a rebuttable presumption 
of “inherent distinctiveness or secondary meaning.” Restatement (Third) of Unfair Competition 
§ 13 cmt. a (1995). Additionally, the Lanham Act provides that if the trademark has been “in 
continuous use for five years subsequent to registration” then the marks become “incontestable,” 

15 U.S.C. § 1065, meaning the registration “shall be conclusive evidence of the validity of the 
registered mark,” including as to whether it is distinctive or has a secondary meaning, 15 U.S.C. 

§ 1115(b); see also Restatement (Third) of Unfair Competition § 13 cmt. a (1995). Plaintiffs 

13 (PSMF 77 (trademark registration for “ASTM”), 78 (trademark registration for “ASTM 
International” and logo), 79 (trademark registration for ASTM logo), 123 (trademark registration 
for “National Fire Protection Association” and “NFPA”), 124 (trademark registration for NFPA 
logo), 126 (trademark registration for NEC logo), 149 (trademark registration for ASHRAE 
logo), 151 (trademark registration for additional ASHRAE logo)). 
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provided evidence that some of their trademarks have become incontestable and that they all are 
distinctive. {See PSMF 77, 78, 124, 125, 126, 150). Defendant offered no evidence to dispute 
the validity of the trademarks. Thus, Plaintiffs have sufficiently established their ownership of 
valid trademarks. 

2. Defendant’s Unauthorized Use in Commerce 

Plaintiffs must also demonstrate that Defendant used their trademarks “in commerce.” 15 
U.S.C. §§1114(1), 1125(a)(1). Under the Lanham Act, ‘“[cjommerce’ means all commerce 
which may be lawfully regulated by Congress.” 15 U.S.C. § 1127. Therefore, to satisfy this 
requirement, Plaintiffs need not demonstrate actual use or intended use in interstate commerce. 

See United We Stand Am., Inc. v. United We Stand, Am. N.Y., Inc., 128 F.3d 86, 92 (2d Cir. 

1997) (the commerce requirement “reflects Congress’s intent to legislate to the limits of its 
authority under the Commerce Clause, rather than to limit the Lanham Act to profit-seeking uses 
of a trademark”). Distribution on the Internet can satisfy the “use in commerce” requirement. 

See Intermatic, Inc. v. Toeppen, 947 F. Supp. 1227, 1239 (N.D. Ill. 1996). Thus, Defendant’s 
online posting of the standards bearing Plaintiffs’ trademarks satisfies this requirement. 

This use in commerce must further be “without the consent of the registrant.” 15 U.S.C. 

§ 1114(1). It is undisputed that Plaintiffs did not authorize Defendant’s use of Plaintiffs’ 
trademarks in commerce. Defendant instead argues that its use was permitted under the “first 
sale doctrine,” which holds that a trademark owner cannot control what happens to its products 
after the first sale. However, the court finds this doctrine a poor fit here, where it is undisputed 
that Defendant did not redistribute the physical copies of Plaintiffs’ standards that it purchased 
but rather created reproductions through scanning and re-typing, with resultant errors and 
differences. See Australian Gold, Inc. v. Hatfield, 436 F.3d 1228, 1241 (10th Cir. 2006) (noting 
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that the first sale doctrine is appropriate only when the actor “does no more than stock, display, 
and resell a producer’s product under the producer’s trademark”); Capitol Records, LLC v. 

DeRigi Inc., 934 F. Supp. 2d 640, 655 (S.D.N.Y. 2013) (in the copyright context, the first sale 
doctrine was “impossible” to apply because that defense is limited to when an actor distributes 
the original material item, not when she distributes reproductions). 

Moreover, Defendant’s quality control standards in reproducing Plaintiffs’ standards 
were outside of Plaintiffs’ control and below that sufficient to deem the standards it distributed 
“genuine” products, meaning the first sale doctrine cannot protect Defendant’s conduct. See 
Polymer Tech. Corp. v. Mimran, 37 F.3d 74, 78 (2d Cir. 1994); Shell Oil Co. v. Commercial 
Petroleum, Inc., 928 F.2d 104, 107 (4th Cir. 1991); El Greco Leather Prods. Co. v. Shoe World, 

806 F.2d 392, 395 (2d Cir. 1986); see also 4 McCarthy on Trademarks and Unfair Competition 
§ 25.42 (4th ed.). Although Defendant argues that there are no material differences between 
Plaintiffs’ standards and Defendant’s reproductions, Plaintiffs need not show that Defendant’s 
reproduced standards were defective, only that they were unable to exercise quality control. See 
Zino DavidoffSA v. CVS Corp., 571 F.3d 238, 243 (2d Cir. 2009). The claim survives because 
“the interference with the trademark holder’s legitimate steps to control quality unreasonably 
subjects the trademark holder to the risk of injury to the reputation of its mark.” Id. Plaintiffs 
have established that Defendant’s quality control standards, including “double-keying” the 
standards, a process involving two separate individuals typing the same material and comparing 
the results to determine the existence of any errors, resulted in missing or inverted pages and 
typographical errors in numerical values or formulas. (ASTM PSMF 190, 214-15). Because 
the standards are therefore not “genuine,” the first sale doctrine does not apply, and Plaintiffs 
have established that Defendant used its trademarks in commerce without authorization. 
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3. Likelihood of Confusion 

Next, the court must assess whether there is a substantial likelihood of consumer 
confusion. This hinges on whether “an appreciable number of ordinarily prudent customers are 
likely to be misled, or simply confused, as to the source” of the copied standards that Public 
Resource posted online. Globalaw, 452 F. Supp. 2d at 47. 

Plaintiffs argue that consumers will be confused both in thinking that Plaintiffs 
authorized Defendant’s posting of the standards, and that Plaintiffs produced the PDF and 
HTML versions of the standards that Defendant posted. See Am Ass ’n for the Advancement of 
Science v. Hearst Corp., 498 F. Supp. 244, 258 (D.D.C. 1980) (noting that both are appropriate 
bases for a confusion argument). Courts in this Circuit consider approximately seven factors in 
assessing the likelihood of confusion, though none is individually determinative. Globalaw, 452 
F. Supp. 2d at 48. They include: (1) the strength of the Plaintiffs’ marks; (2) the degree of 
similarity between the marks; (3) the proximity of the products; (4) evidence of actual confusion; 

(5) Defendant’s purpose or reciprocal good faith in adopting its own mark; (6) the quality of 
Defendant’s product; and (7) the sophistication of the buyers. Id. Several courts in other 
Circuits have determined that when a defendant uses an identical mark on a similar product, 
consideration of all the factors is not necessary. Seelnt’l Cosmetics Exch., Inc. v. Gapardis 
Health & Beauty, Inc., 303 F.3d 1242, 1248-49 (11th Cir. 2002); Wynn Oil Co. v. Thomas, 839 
F.2d 1183, 1190-91 (6th Cir. 1988). 

Defendant does not dispute that Plaintiffs’ marks are “strong,” that Defendant used marks 
and logos that are identical to Plaintiffs’ marks and logos when it posted the Plaintiffs’ standards 
online, and that the standards it applied the marks and logos to were identical or nearly identical 
to Plaintiffs’. (PSMF fflf 210-11; Def. Br. at 65). Moreover, it is undisputed that the standards 
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distributed by Plaintiffs and by Defendant were in close proximity, since Defendant offered the 
standards in the same market as Plaintiff—i.e., the Internet—as a free alternative to purchasing 
the standards from Plaintiffs directly. See Restatement (Third) of Unfair Competition § 21 cmt. j 
(1995) (“[T]he use of similar designations on goods that are used together, or that perform the 
same function, or that are of the same general class, is more likely to cause confusion than is a 
use in connection with goods used for different purposes, or in different contexts, or by different 
purchasers.”). It is also undisputed that Defendant intended for individuals to consider that the 
standards were identical. (PSMF1213). 

Defendant argues that despite these undisputed facts, consumers would not be confused 

because it posts disclaimers that it claims “adequately informed consumers” so that “no 

reasonable consumer would mistake [its cover page] as part of the original document.” (Def. 

Reply at 28 (referring to the PDF disclaimer at ASTM ECF No. 118-12, Ex. 16)). Defendant 

also argues that the PDF versions it posted “look like scans of physical documents,” and that the 

“preamble for the .html standards informs reasonable consumers that Public Resource has 

provided the transcription.” {Id. (referring to the HTML disclaimer at ASTM ECF No. 118-13, 

Ex. 26)). 14 Here, Defendant’s disclaimer on the PDF reads in full: 

In order to promote public education and public safety, equal justice for all, a 
better informed citizenry, the rule of law, world trade and world peace, this legal 
document is hereby made available on a noncommercial basis, as it is the right of 
all humans to know and speak the laws that govern them. 

(ASTM ECF No. 118-12, Ex. 16). The disclaimer on the HTML versions contain s similar 


14 Defendant cites to Prestonettes, Inc. v. Coty, 264 U.S. 359, 369 (1924), in support of its 
argument that a disclaimer is sufficient to inform consumers that it has repackaged or changed 
the original. The facts of that case do not support Defendant’s position, as the disclaimer in that 
case stated clearly that the distributor was not connected with the producer and that the 
producer’s product was merely a constituent part of the distributor’s new product. Coty, 264 
U.S. at 367. 
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language. (ASTMECFNo. 118-13, Ex. 26). These disclaimers do not mention Defendant’s 
creation of the reproductions, Plaintiffs’ lack of association or authorization, or that they are even 
reproductions or transcriptions, and can hardly be called disclaimers at all. Moreover, 

Defendant’s assertion that the PDFs “look like scans” offers no assistance to a consumer looking 
at the standard, as they would have no way to determine whether the Plaintiffs or Defendant 
created the scan. While Defendant has since adopted a more thorough disclaimer that includes 
information about Public Resource’s retyping of the HTML versions and the possibility of errors 
(DSMF ]f 169), it did not begin using that disclaimer until 2015, after the start of this litigation. 
(Decl. of Carl Malamud 131 (ASTM ECF No. 122-8)). 

The parties have presented no evidence to establish the existence or non-existence of 
actual consumer confusion. While such evidence is not required, without it summary judgment 
on consumer confusion, and trademark infringement more generally, is a difficult call. However, 
the facts here present nearly as black-and-white a case as possible. A consumer in the market for 
one of Plaintiffs’ voluntary consensus standards may encounter them on Plaintiffs’ websites for 
purchase, or on Defendant’s website for free download. Because Defendant has intentionally 
created a copy that is meant to appear identical, including use of Plaintiffs’ trademarks, then that 
consumer may download that standard for free from Defendant without knowing that it is not 
created by the Plaintiffs and may contain missing pages or typographical errors leading to 
inaccurate values for measurements. In short, Plaintiffs have presented enough evidence for the 
court to conclude that there is no genuine dispute on the factual issue of whether consumer 
confusion is likely. 

4. Defendant’s Nominative Fair Use Defense 

While Plaintiffs have successfully established Defendant’s infringing use of their 
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trademarks, Defendant argues that its use of Plaintiffs’ trademarks is “nominative fair use.” 

Under this defense, Defendant must demonstrate that its use of Plaintiffs’ trademarks was 
necessary to describe their standards; that it only used as much of the marks as was reasonably 
necessary to identify the standards; and that it has not done anything to suggest sponsorship or 
endorsement by the Plaintiffs or to inaccurately describe the relationship between the parties’ 
products. See Rosetta Stone Ltd. v. Google, Inc., 676 F.3d 144, 154 (4th Cir. 2012). Nominative 
fair use by a defendant makes it “clear to consumers that the plaintiff, not the defendant, is the 
source of the trademarked product or service.” Century 21 Real Estate Corp. v. Lendingtree, 

Inc., 425 F.3d 211, 220 (3d Cir. 2005). Thus, if Defendant’s use is nominative fair use, it would 
not create “confusion about the source of [the] defendant’s product.” Tiffany (NJ) Inc. v. eBay 
Inc., 600 F.3d 93, 102 (2d Cir. 2010) (alteration in original). On this point, the parties argue past 
each other. Defendant believes no consumer would believe that Defendant, rather than Plaintiffs, 
was the source of the standards, and so its use is a fair use. Plaintiffs argue that Defendant’s use 
ca nn ot be fair precisely because consumers would believe that Plaintiffs were the source of the 
reproduced standards, which they are not. However, because the court has already determined 
that consumer confusion as to the source of the trademarked standards is likely, the nominative 
fair use defense is inapplicable and the court need not assess each of the Rosetta Stone factors 
listed above. 

The court therefore finds that Defendant engaged in trademark infringement by its use of 
Plaintiffs’ registered trademarks, and Plaintiffs’ motion for summary judgment on their 
trademark claims is GRANTED and Defendant’s cross-motion is DENIED. 

IV. REMEDIES 

Both ASTM Plaintiffs and AERA Plaintiffs seek a permanent injunction barring 
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Defendant from distributing, displaying, or creating derivative works from their copyrighted 
standards and, in the case of ASTM Plaintiffs, their trademarks, which this court has authority to 
grant under 17 U.S.C. § 502(a) (Copyright Act) and 15 U.S.C. § 1116 (Lanham Act). Plaintiffs 
must establish (1) irreparable injury; (2) that remedies available at law, such as monetary 
damages, are inadequate to compensate for their injury; (3) that a remedy in equity is warranted 
after considering the balance of hardships; and (4) that the public interest would not be disserved 
by a permanent injunction. See eBay Inc. v. MercExchange, LLC, 547 U.S. 388, 391 (2006). 

A. Irreparable Injury 

The ASTM Plaintiffs assert that they will face three separate irreparable injuries if 
Defendant is permitted to continue distribution of Plaintiffs’ standards, including substantial 
declines in revenue that may cause their business models to change, the loss of the exclusive 
rights under the Copyright Act to exclude others from distributing, reproducing, or displaying 
their protected works, and the loss of control of the goodwill associated with their trademarks. 
AERA Plaintiffs similarly assert that they will face three separate irreparable injuries if 
Defendant is permitted to continue distribution of Plaintiffs’ standards, including loss of business 
opportunities, the loss of the exclusive rights under the Copyright Act to exclude others from 
distributing, reproducing, or displaying their protected works, and the adverse effect on 
Plaintiffs’ efforts to create further standards. 

It is well established that the threat of continuing copyright infringement justifies 
granting a permanent injunction. See Walt Disney Co. v. Powell, 897 F.2d 565, 567 (D.C. Cir. 
1990) (“When a [ ] plaintiff has established a threat of continuing infringement, he is entitled to 
an injunction.”); Hanley-Wood LLC v. Hanley Wood LLC, 783 F. Supp. 2d 147, 151 (D.D.C. 

2011); Breaking the Chain Found, v. Capital Educ. Support, Inc., 589 F. Supp. 2d 25, 30 
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(D.D.C. 2008). While a court should not automatically issue an injunction after it finds there 
was past copyright or trademark infringement, here Plaintiffs’ alleged irreparable injury is not 
the past infringement but the threat of future infringement. Defendant has not provided any 
assurances that it would cease posting of Plaintiffs’ standards—indeed, it is undisputed that 
during the course of this litigation, Public Resource posted online versions of the ASTM 
Plaintiffs’ other standards not involved in this litigation. (PSMF 235). Moreover, Defendant’s 
counsel at oral argument admitted that Defendant would post the AERA Plaintiffs’ 2014 
Standards if they were incorporated by reference into federal regulations in the future. (Tr. of 
Motions Hearing at 75:24—76:2). The court thus determines that the continued threat of 
infringement is sufficient to weigh in favor of an injunction. 

B. Adequacy of Monetary Damages 

Plaintiffs argue that because damages here are difficult to quantify and Defendant may be 
unable to pay damages, then legal remedies are inadequate. See Fox Television Stations, Inc. v. 
FilmOn XLLC, 966 F. Supp. 2d. 30, 50 (D.D.C. 2013). The evidence shows that while the 
Plaintiffs’ standards were accessed thousands of times on Defendant’s website, Defendant does 
not track information that would be helpful in calculating damages, such as how many of those 
accesses actually led to downloads, and whether those downloads were in lieu of purchases. 
Moreover, Defendant did not dispute that it has “extremely limited financial resources available 
to pay any damages award” and that in 2014 it “generated under $100,000 in operating income 
and had $248,000 in total net assets.” (ASTM PSMF fflf 272-73). Given that the Copyright Act 
provides for statutory damages ranging from $750 to $30,000 for each of the standards at issue in 
the overall case, or even up to $150,000 per infringement if Plaintiffs were to later prove that 
infringement was committed willfully, Defendant’s potential inability to pay is surely a factor 
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weighing towards equitable relief. See 17 U.S.C. § 504(c)(l)-(2). 

C. Balance of Hardships & Public Interest 

The court must weigh the likely harms faced by Plaintiffs described above with any 
harms faced by Defendant if an injunction is imposed. Here, Defendant’s CEO Carl Malamud 
was asked in his ASTM deposition what financial impact an injunction barring posting of the 
standards would have on Public Resource, and he responded “probably none.” (Malamud Dep. 
at 219:22-220:4 (Ex. 3 to Rubel Deck (ASTM ECF No. 118-12))). The only harm Mr. Malamud 
identified was that “one hates to have wasted that [] effort” that went into posting the standards 
online. (Id.). Without evidence of any additional harms, this factor weighs strongly in favor of 
an injunction. 

Additionally, the public must not be disserved by the issuance of an injunction. Here, the 
public interest is served by the policy interests that underlie the Copyright Act itself, namely the 
protection of financial incentives for the continued creation of valuable works, and the continued 
value in maintaining the public-private system in place in the U.S. to ensure continued 
development of technical standards. 

Taken together, the court finds that injunctive relief is appropriate and that Defendant 
should be permanently barred from violating any of Plaintiffs’ exclusive copyrights, including 
distributing, displaying, reproducing, or creating derivative works in the nine standards on which 
ASTM Plaintiffs moved for summary judgment and AERA Plaintiffs’ 1999 Standards, as well as 
barred from any use of ASTM Plaintiffs’ trademarks in connection with the posting of these 
standards online or elsewhere. 

V. CONCLUSION 

For the reasons set forth above, ASTM Plaintiffs’ Motion is GRANTED, AERA 
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Plaintiffs’ Motion is GRANTED IN PART and DENIED IN PART, and Defendant’s Cross- 
Motions are DENIED. 


Date: February 2, 2017 






TANYA S. CHUTKAN 


United States District Judge 
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UNITED STATES DISTRICT COURT 
FOR THE DISTRICT OF COLUMBIA 


AMERICAN SOCIETY FOR TESTING 
AND MATERIALS, et al.. 

Plaintiffs, 

v. 

PUBLIC.RESOURCE.ORG, INC., 
Defendant. 


) 

) 

) 

) 

) 

) Case No. 13-cv-1215 (TSC) 

) 

) 

) 

) 

) 

) 

ORDER 


Upon consideration of the parties’ motions, and for the reasons set forth in the court’s 
Memorandum Opinion, Plaintiffs’ motion is GRANTED and Defendant’s motion is DENIED. 


It is hereby ORDERED that Defendant is permanently enjoined from all unauthorized 
use, including through reproduction, display, distribution, or creation of derivative works, of the 
following nine standards: ASTM D86-07, ASTM D975-07, ASTM D396-98, ASTM D1217- 
93(98), NFPA National Electrical Code, 2011 ed. and 2014 ed., ASHRAE Standard 90.1, 2004 
ed., 2007 ed., and 2010 ed. It is FURTHER ORDERED that Defendant is permanently enjoined 
from all unauthorized use of Plaintiffs’ registered trademarks. 

Defendant is FURTHER ORDERED to remove all versions of these nine standards from 
its website and any other website within its possession, custody, or control within five days. 


Date: February 2, 2017 


S-. 




TANYA S. CHUTKAN 


United States District Judge 
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IN THE UNITED STATES DISTRICT COURT 
FOR THE DISTRICT OF COLUMBIA 


AMERICAN SOCIETY FOR TESTING AND 
MATERIALS d/b/a ASTM INTERNATIONAL; 


Case No. l:13-cv-01215-TSC-DAR 


NATIONAL FIRE PROTECTION 
ASSOCIATION, INC.; and 

AMERICAN SOCIETY OF HEATING, 
REFRIGERATING, AND AIR CONDITIONING 
ENGINEERS, 


Plaintiffs/Counter-defendants, 


NOTICE OF APPEAL BY 
DEFENDANT-COUNTERCLAIMANT 
PUBLIC.RESOURCE.ORG, INC. 


Action Filed: August 6, 2013 


PUBLIC.RESOURCE.ORG, INC., 


Defendant/Counterclaimant. 


Pursuant to 28 U.S.C. § 1292(a)(1), Defendant and Counterclaimant Public.Resource.Org, 
Inc. hereby gives notice of its appeal to the United States Court of Appeals for the District of 
Columbia Circuit from the order of February 2, 2017, Dkt. no. 176, permanently enjoining 
Public.Resource.Org and granting Plaintiffs’ Motion for Summary Judgment and for a Permanent 
Injunction, pursuant to this Court’s decision in the memorandum opinion of the same date, Dkt. no. 
175. 
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Dated: February 15, 2017 


Respectfully submitted, 

(s/_Andrew P. Bridges _ 

Andrew P. Bridges (admitted) 

abridges@fenwick. com 

Sebastian E. Kaplan (admitted pro hac vice) 

skaplan@fenwick. com 

Matthew B. Becker (admitted pro hac vice) 

mbecker@fenwick. com 

FENWICK & WEST LLP 

555 California Street, 12th Floor 

San Francisco, CA 94104 

Telephone: (415) 875-2300 

Facsimile: (415)281-1350 

Mitchell L. Stoltz (D.C. Bar No. 978149) 
mitch@eff.org 

Corynne McSherry (admitted pro hac vice) 
cor 5 mne@eff. 0 rg 

ELECTRONIC FRONTIER FOUNDATION 

815 Eddy Street 

San Francisco, CA 94109 

Telephone: (415) 436-9333 

Facsimile: (415)436-9993 

David Halperin (D.C. Bar No. 426078) 
davidhalperindc@gmail.com 
1530 P Street NW 
Washington, DC 20005 
Telephone: (202) 905-3434 

Attorneys for Defendant-Counterclaimant 
Public.Resource. Org, Inc. 
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CERTIFICATE OF SERVICE 


I hereby certify that the foregoing was electronically filed with the Clerk of the Court for 
the District of Columbia and served on all counsel of record via the CM/ECF system on 
February 15, 2017. 


/s/ Andrew P. Bridges 

Andrew P. Bridges 
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UNITED STATES DISTRICT COURT 
FOR THE DISTRICT OF COLUMBIA 


) 

AMERICAN SOCIETY FOR TESTING ) 

AND MATERIALS, et al., ) 

) 

Plaintiffs, ) 

) Case No. 13-cv-1215 (TSC) 
v. ) 

) 

PUBLIC.RESOURCE.ORG, INC., ) 

) 

Defendant. ) 

_ ) 

AMENDED ORDER 

Upon consideration of the parties’ joint stipulation to a modified order in their Joint 
Status Report (ECF No. 179), the court hereby amends its February 2, 2017 Order (ECF No. 

176) to include the following: 

It is ORDERED that Defendant is permanently enjoined from all unauthorized use of 
Plaintiffs’ registered trademarks, provided that nothing in this injunction is intended to or does 
apply to (a) Defendant’s use of Plaintiffs’ names or logos in those documents that have been 
filed publicly in this action and that Defendant has posted to its website; (b) Defendant’s posting 
or other use of letters that Plaintiffs themselves have sent to Defendant, governmental agencies, 
or third parties and that include Plaintiffs’ names or logos; (c) Defendant’s use of Plaintiffs’ 
names or the names of their standards in tables that Defendant has posted to its website to 
indicate jurisdictions that have incorporated such standards by reference; or (d) Defendant’s 
reference to Plaintiffs’ names or the names of their standards in Defendant’s critical commentary 
and political speech, subject to the limitation that this does not permit Public Resource to display 
on its website Plaintiffs’ registered trademarks in versions of the standards that Plaintiffs publish. 


1 


JA2118 





Case l:13-cv-01215-TSC Document 182 Filed 04/03/17 Page 2 of 2 
USCA Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 360 of 573 


Moreover, this injunction does not apply to third-party standards posted by Public Resource that 
merely reference Plaintiffs or Plaintiffs’ works without reproducing Plaintiffs’ works in whole or 
substantial part. 


Date: April 3, 2017 


S-. 




TANYA S. CHUTKAN 


United States District Judge 
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IN THE UNITED STATES DISTRICT COURT 
FOR THE DISTRICT OF COLUMBIA 


AMERICAN SOCIETY FOR TESTING AND 
MATERIALS d/b/a ASTM INTERNATIONAL; 


Case No. l:13-cv-01215-TSC-DAR 


NATIONAL FIRE PROTECTION 
ASSOCIATION, INC.; and 

AMERICAN SOCIETY OF HEATING, 
REFRIGERATING, AND AIR CONDITIONING 
ENGINEERS, 


Plaintiffs/Counter-defendants, 


AMENDED NOTICE OF APPEAL BY 
DEFENDANT-COUNTERCLAIMANT 
PUBLIC.RESOURCE.ORG, INC. 


Action Filed: August 6, 2013 


PUBLIC.RESOURCE.ORG, INC., 


Defendant/Counterclaimant. 


Pursuant to 28 U.S.C. § 1292(a)(1), Defendant and Counterclaimant Public.Resource.Org, 
Inc. hereby gives amended notice of its appeal to the United States Court of Appeals for the 
District of Columbia Circuit from the Order of February 2, 2017, Dkt. no. 176, to reflect and 
include the Court’s Amended Order issued April 3, 2017, Dkt. no. 182, permanently enjoining 
Public.Resource.Org and granting Plaintiffs’ Motion for Summary Judgment and for a Permanent 
Injunction, pursuant to this Court’s decision in the memorandum opinion of February 2, 2017, 
Dkt. no. 175. 


JA2120 




Case l:13-cv-01215-TSC Document 183 Filed 04/06/17 Page 2 of 3 
USCA Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 362 of 573 


Dated: April 6 , 2017 


Respectfully submitted, 

(s/_Andrew P. Bridges _ 

Andrew P. Bridges (admitted) 

abridges@fenwick. com 

Sebastian E. Kaplan (admitted pro hac vice) 

skaplan@fenwick. com 

Matthew B. Becker (admitted pro hac vice) 

mbecker@fenwick. com 

FENWICK & WEST LLP 

555 California Street, 12th Floor 

San Francisco, CA 94104 

Telephone: (415) 875-2300 

Facsimile: (415)281-1350 

Mitchell L. Stoltz (D.C. Bar No. 978149) 
mitch@eff.org 

Corynne McSherry (admitted pro hac vice) 
cor 5 mne@eff. 0 rg 

ELECTRONIC FRONTIER FOUNDATION 

815 Eddy Street 

San Francisco, CA 94109 

Telephone: (415) 436-9333 

Facsimile: (415)436-9993 

David Halperin (D.C. Bar No. 426078) 
davidhalperindc@gmail.com 
1530 P Street NW 
Washington, DC 20005 
Telephone: (202) 905-3434 

Attorneys for Defendant-Counterclaimant 
Public.Resource. Org, Inc. 
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CERTIFICATE OF SERVICE 


I hereby certify that the foregoing was electronically filed with the Clerk of the Court for 
the District of Columbia and served on all counsel of record via the CM/ECF system on 
April 6, 2017. 


/s/ Andrew P. Bridges 

Andrew P. Bridges 
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APPEAL,CLOSED,STAYED,TYPE-E 


U.S. District Court 

District of Columbia (Washington, DC) 
CIVIL DOCKET FOR CASE #: l:14-cv-00857-TSC 


AMERICAN EDUCATIONAL RESEARCH Date Filed: 05/23/2014 

ASSOCIATION, INC. et al v. PUBLIC.RESOURCE.ORG, Date Terminated: 07/10/2017 
INC. Jury Demand: Defendant 

Assigned to: Judge Tanya S. Chutkan Nature of Suit: 820 Copyright 

Case: l:13-cv-01215-TSC Jurisdiction: Federal Question 

Case in other court: USCA, 17-07039 
Cause: 17:101 Copyright Infringement 

Plaintiff 

AMERICAN EDUCATIONAL represented by Clifton Scott Elgarten 

RESEARCH ASSOCIATION, INC. CROWELL & MORING LLP 

1001 Pennsylvania Avenue, NW 
Suite 1100 

Washington, DC 20004-2595 
(202) 624-2523 
Fax: (202) 628-5116 
Email: celgarten@crowell.com 
LEAD ATTORNEY 
ATTORNEY TO BE NOTICED 

Jonathan Hudis 

QUARLES & BRADY LLP 
Intellectual Property Practice Group 
1701 Pennsylvania Avenue, NW 
Suite 700 

Washington, DC 20006 
(202) 372-9528 
Fax: (202) 372-9599 
Email: jon.hudis@quarles.com 
TERMINATED: 03/24/2017 

Jonathan P. Labukas 

QUARLES & BRADY, LLP 

1701 K Pennsylvania Avenue, NW 

Washington, DC 20006 

(202) 372-9514 

Fax: (202) 372-9594 

Email: jonathan.labukas@quarles.com 

TERMINATED: 03/24/2017 

Kathleen Cooney-Porter 
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OBLON, MCCLELLAND, MAIER & 

NEUSTADT, LLP 

1940 Duke Street 

Alexandria, VA 22314 

(703)413-3000 

Email: kcooney-porter@oblon.com 
TERMINATED: 01/21/2016 

Nikia L. Gray 

QUARLES & BRADY, LLP 
1701 K Pennsylvania Avenue, NW 
Washington, DC 20006 
(202) 372-9600 
Fax: (202) 372-9599 
Email: nikia.gray@quarles.com 
TERMINATED: 03/24/2017 
PRO HAC VICE 

Plaintiff 

represented by Clifton Scott Elgarten 
(See above for address) 

LEAD ATTORNEY 
ATTORNEY TO BE NOTICED 

Jonathan Hudis 

(See above for address) 

TERMINATED: 03/24/2017 

Jonathan P. Labukas 

(See above for address) 

TERMINATED: 03/24/2017 

Kathleen Cooney-Porter 

(See above for address) 

TERMINATED: 01/21/2016 

Nikia L. Gray 

(See above for address) 

TERMINATED: 03/24/2017 
PRO HAC VICE 


Plaintiff 

NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION 
INC. 


Jonathan Hudis 

(See above for address) 


represented by Clifton Scott Elgarten 
(See above for address) 

LEAD ATTORNEY 
A TTORNEY TO BE NOTICED 


AMERICAN PSYCHOLOGICAL 
ASSOCIATION, INC. 
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Jonathan P. Labukas 

(See above for address) 
TERMINATED: 03/24/2017 

Kathleen Cooney-Porter 

(See above for address) 
TERMINATED: 01/21/2016 

Nikia L. Gray 

(See above for address) 
TERMINATED: 03/24/2017 
PRO HAC VICE 
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TERMINATED: 03/24/2017 


V. 

Defendant 

PUBLIC.RESOURCE.ORG, INC. represented by Andrew Phillip Bridges 

FENWICK & WEST, LLP 
555 California Street 
Suite 1200 

San Francisco, CA 94104 
(415) 875-2389 

Email: abridges@fenwick.com 
LEAD ATTORNEY 
ATTORNEY TO BE NOTICED 


Mitchell L. Stoltz 

ELECTRONIC FRONTIER 
FOUNDATION 
815 Eddy Street 
San Francisco, CA 94109 
(415) 436-9333 
Fax: (415) 436-9993 
Email: mitch@eff.org 
LEAD ATTORNEY 
ATTORNEY TO BE NOTICED 


Sebastian E. Kaplan 

FENWICK & WEST LLP 
555 California Street 
12th Floor 

San Francisco, CA 94104 
(415) 875-2300 
LEAD ATTORNEY 
PRO HAC VICE 
ATTORNEY TO BE NOTICED 
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Corynne McSherry 

ELECTRONIC FRONTIER 
FOUNDATION 
815 Eddy Street 
San Francisco, CA 94109 
(415) 436-9333 
Fax: (415) 436-9993 
Email: cor 3 mne@eff. 0 rg 
PRO HAC VICE 
ATTORNEY TO BE NOTICED 

David Elliot Halperin 

1530 P Street, NW 
Washington, DC 20005 
(202) 905-3434 

Email: davidhalperindc@gmail.com 
ATTORNEY TO BE NOTICED 


Matthew B. Becker 

FENWICK & WEST LLP 
801 California Street 
Mountain View, CA 94041 
(650)335-7930 

Email: mbecker@fenwick.com 
PRO HAC VICE 
ATTORNEY TO BE NOTICED 


Amicus 

REPORTERS COMMITTEE FOR represented by Bruce D. Brown 

FREEDOM OF THE PRESS REPORTERS COMMITTEE FOR 

FREEDOM OF THE PRESS 
1156 15th St. NW 
Suite 1250 

Washington, DC 20005 
(202) 795-9301 
Fax: (202) 795-9310 
Email: bbrown@rcfp.org 
LEAD ATTORNEY 
ATTORNEY TO BE NOTICED 


Amicus 

LAW SCHOLARS represented by Catherine R. Gellis 

P.O. Box 2477 
Sausalito, CA 94966 
(202) 642-2849 
Email: cathy@cgcounsel.com 
LEAD ATTORNEY 
ATTORNEY TO BE NOTICED 
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Amicus 

PUBLIC KNOWLEDGE represented by Charles Duan 

PUBLIC KNOWLEDGE 
1818 N Street, NW 
Suite 410 

Washington, DC 20036 
(202) 861-0020x 119 
Email: cduan@publicknowledge.org 
LEAD ATTORNEY 
ATTORNEY TO BE NOTICED 

Amicus 

SINA BAHRAM represented by Jeffrey T. Pearlman 

MILLS LEGAL CLINIC AT 
STANLORD LAW SCHOOL 
559 Nathan Abbott Way. 

Stanford, CA 94305 
(650) 497-9443 
Lax: (650) 723-4426 
Email: jef@law.stanford.edu 
LEAD ATTORNEY 
ATTORNEY TO BE NOTICED 


V. 

Intervenor 

AMERICAN SOCIETY FOR represented by J. Kevin Fee 

TESTING AND MATERIALS MORGAN, LEWIS & BOCKUS LLP 

100 BARR HARBOR DRIVE 1111 Pennsylvania Avenue, N.W. 

WEST CONSHOHOCKEN, PA 19428 Washington, DC 20004 

(202) 739-5353 
Fax: (202) 239-3001 
Email: jkfee@morganlewis.com 
LEAD ATTORNEY 
ATTORNEY TO BE NOTICED 

Intervenor 

NATIONAL FIRE PROTECTION represented by Anjan Choudhury 

ASSOCIATION, INC. MUNGER, TOLLES & OLSON LLP 

350 South Grand Avenue 
50th Floor 

Los Angeles, CA 90071 

(212) 683-9107 

Fax:(213) 683-5107 

Email: anjan.choudhury@mto.com 

LEAD ATTORNEY 

ATTORNEY TO BE NOTICED 
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Intervenor 

AMERICAN SOCIETY OF represented by Joseph R. Wetzel 

HEATING, REFRIGERATING, KING & SPALDING, LLP 

AND AIR-CONDITIONING 101 2nd Street 

ENGINEERS, INC. Suite 2300 

San Francisco, CA 94105 
(415)318-1200 
Fax: (415)318-1300 
Email: jwetzel@kslaw.com 
LEAD ATTORNEY 
ATTORNEY TO BE NOTICED 

Counter Claimant 

PUBLIC.RESOURCE.ORG, INC. represented by Andrew Phillip Bridges 

(See above for address) 

LEAD ATTORNEY 
ATTORNEY TO BE NOTICED 

Mitchell L. Stoltz 

(See above for address) 

LEAD ATTORNEY 
ATTORNEY TO BE NOTICED 

Corynne McSherry 

(See above for address) 

PRO HAC VICE 
ATTORNEY TO BE NOTICED 

David Elliot Halperin 

(See above for address) 
ATTORNEY TO BE NOTICED 

Matthew B. Becker 

(See above for address) 
ATTORNEY TO BE NOTICED 


V. 

Counter Defendant 

AMERICAN EDUCATIONAL represented by Clifton Scott Elgarten 

RESEARCH ASSOCIATION, INC. (See above for address) 

LEAD ATTORNEY 
ATTORNEY TO BE NOTICED 


Jonathan Hudis 

(See above for address) 
TERMINATED: 03/24/2017 
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Jonathan P. Labukas 

(See above for address) 

TERMINATED: 03/24/2017 

Kathleen Cooney-Porter 

(See above for address) 

TERMINATED: 01/21/2016 

Nikia L. Gray 

(See above for address) 

TERMINATED: 03/24/2017 
PRO HAC VICE 


Counter Defendant 

represented by Clifton Scott Elgarten 
(See above for address) 

LEAD ATTORNEY 
ATTORNEY TO BE NOTICED 

Jonathan Hudis 

(See above for address) 
TERMINATED: 03/24/2017 

Jonathan P. Labukas 

(See above for address) 
TERMINATED: 03/24/2017 

Kathleen Cooney-Porter 

(See above for address) 
TERMINATED: 01/21/2016 

Nikia L. Gray 

(See above for address) 
TERMINATED: 03/24/2017 
PRO HAC VICE 


Counter Defendant 

NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION 
INC. 


Jonathan Hudis 

(See above for address) 
TERMINATED: 03/24/2017 

Jonathan P. Labukas 

(See above for address) 
TERMINATED: 03/24/2017 


represented by Clifton Scott Elgarten 

(See above for address) 

LEAD ATTORNEY 
ATTORNEY TO BE NOTICED 


AMERICAN PSYCHOLOGICAL 
ASSOCIATION, INC. 
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Kathleen Cooney-Porter 

(See above for address) 
TERMINATED: 01/21/2016 

Nikia L. Gray 

(See above for address) 
TERMINATED: 03/24/2017 
PRO HAC VICE 


Date Filed 

# 

Docket Text 

05/23/2014 

1 

COMPLAINT filed against PUBLIC.RESOURCE.ORG, INC. (Filing fee $ 

400 receipt number 0090-3725541) filed by AMERICAN EDUCATIONAL 
RESEARCH ASSOCIATION, INC., NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC., AMERICAN 

PSYCHOLOGICAL ASSOCIATION, INC.. (Attachments: # ! Exhibit A, #2 
Exhibit B, # 3 Coiporate Disclosure Stmt, # 4 Civil Cover Sheet, # 5 

Summons, # 6 Report to Register of Copyrights)(Hudis, Jonathan) (Entered: 
05/23/2014) 

05/23/2014 


Case Assigned to Judge Gladys Kessler, (kb) (Entered: 05/27/2014) 

05/27/2014 

2 

SUMMONS (1) Issued Electronically as to PUBLIC.RESOURCE.ORG, INC. 
(Attachments: # l Consent Form, # 2 Notice of Consent)(kb) (Entered: 
05/27/2014) 

05/28/2014 

3 

Case randomly reassigned to Judge Christopher R. Cooper. Judge Gladys 

Kessler no longer assigned to the case, (gt,) (Entered: 05/28/2014) 

06/09/2014 

4 

NOTICE of Appearance by Kathleen Cooney-Porter on behalf of AMERICAN 
EDUCATIONAL RESEARCH ASSOCIATION, INC., AMERICAN 
PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC. (Cooney-Porter, Kathleen) 

(Entered: 06/09/2014) 

06/10/2014 

5 

RETURN OF SERVICE/AFFIDAVIT of Summons and Complaint Executed. 
PUBLIC.RESOURCE.ORG, INC. served on 6/2/2014, answer due 6/23/2014 
(Hudis, Jonathan) (Entered: 06/10/2014) 

07/01/2014 


MINUTE ORDER: Upon review of the record, it appears that the time for 
defendant Public.Resource.Org, Inc. to answer or otherwise respond to 
plaintiffs' complaint has expired. Plaintiffs are therefore ORDERED to show 
cause by July 15, 2014, as to why they have failed to move for default 
judgment and why the case should not be dismissed for want of prosecution. 
Signed by Judge Christopher R. Cooper on 7/1/2014. (Iccrc2, ) (Entered: 
07/01/2014) 

07/07/2014 

6 

NOTICE of Appearance by David Elliot Halperin on behalf of 
PUBLIC.RESOURCE.ORG, INC. (Halperin, David) (Entered: 07/07/2014) 

07/07/2014 

7 
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NOTICE of Appearance by Mitchell L. Stoltz on behalf of 
PUBLIC.RESOURCE.ORG, INC. (Stoltz, Mitchell) (Entered: 07/07/2014) 

07/07/2014 

8 

LCvR 7.1 CERTIFICATE OF DISCLOSURE of Corporate Affiliations and 
Financial Interests by PUBLIC.RESOURCE.ORG, INC. (Stoltz, Mitchell) 
(Entered: 07/07/2014) 

07/07/2014 

9 

Unopposed MOTION for Extension of Time to File Answer to Complaint by 
PUBLIC.RESOURCE.ORG, INC. (Attachments: # 1 Text of Proposed Order) 
(Stoltz, Mitchell) (Entered: 07/07/2014) 

07/08/2014 


MINUTE ORDER granting 9 Unopposed Motion for Extension of Time. 
Defendant shall file its response to the Complaint on or before July 14, 2014. 

The parties are reminded that the Court will only giant extensions of time upon 
the filing of a motion. Signed by Judge Christopher R. Cooper on 7/8/2014. 
(Iccrc2,) (Entered: 07/08/2014) 

07/08/2014 

10 

NOTICE of Appeaiance by Andrew Phillip Bridges on behalf of 
PUBLIC.RESOURCE.ORG, INC. (Bridges, Andrew) (Entered: 07/08/2014) 

07/09/2014 

11 

RESPONSE TO ORDER TO SHOW CAUSE by AMERICAN 
EDUCATIONAL RESEARCH ASSOCIATION, INC., AMERICAN 
PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC. as noted in Court's Minute Order 
of July 1, 2014. (Hudis, Jonathan) (Entered: 07/09/2014) 

07/14/2014 

12 

ANSWER to Complaint with Jury Demand, COUNTERCLAIM FOR 
DECLARATORY RELIEF against All Plaintiffs by 

PUBLIC.RESOURCE.ORG, INC..(Bridges, Andrew) (Entered: 07/14/2014) 

07/23/2014 

11 

Unopposed MOTION for Extension of Time to File Response/Reply to 
Defendant's Counterclaim by AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, 
INC., NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION, INC. 
(Attachments: #1 Text of Proposed Order)(Hudis, Jonathan) (Entered: 
07/23/2014) 

07/23/2014 


MINUTE ORDER granting J3 Plaintiffs' Unopposed Motion for Extension of 
Time. Plaintiffs shall file their answer or other response to Defendant's 
Counterclaim on or before August 25, 2014. Signed by Judge Christopher R. 
Cooper on 7/23/2014. (Iccrc2,) (Entered: 07/23/2014) 

08/21/2014 

14 

Plaintiffs' ANSWER to 12 Answer to Complaint, COUNTERCLAIM filed by 
Defendant for Declaratory Relief by AMERICAN EDUCATIONAL 
RESEARCH ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL 
ASSOCIATION, INC., NATIONAL COUNCIL ON MEASUREMENT IN 
EDUCATION, INC.. Related document: 12 Answer to Complaint, 
COUNTERCLAIM filed by PUBLIC.RESOURCE.ORG, INC..(Hudis, 
Jonathan) (Entered: 08/21/2014) 

08/21/2014 

15 

MOTION to Strike 12 Answer to Complaint, COUNTERCLAIM re 

Defendant's Jury Demand by AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, 
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INC., NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION, INC. 
(Attachments: # J. Text of Proposed Order Striking Defendant's Jmy Demand) 
(Hudis, Jonathan) (Entered: 08/21/2014) 

09/08/2014 

16 

Memorandum in opposition to re 15 MOTION to Strike 12 Answer to 
Complaint, COUNTERCLAIM re Defendant's Jury Demand filed by 
PUBLIC.RESOURCE.ORG, INC.. (Attachments: # J. Text of Proposed Order) 
(Stoltz, Mitchell) (Entered: 09/08/2014) 

09/18/2014 

12 

REPLY to opposition to motion re _1_5 MOTION to Strike 12 Answer to 
Complaint, COUNTERCLAIM re Defendant's Jury Demand filed by 
AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC., 
AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL 
COUNCIL ON MEASUREMENT IN EDUCATION, INC.. (Hudis, Jonathan) 
(Entered: 09/18/2014) 

09/18/2014 

18 

ORDER FOR INITIAL SCHEDULING CONFERENCE: The above-captioned 
case has been assigned to this Judge for resolution. It is hereby ORDERED that 
the Initial Scheduling Conference be set for Thursday, October 16, 2014 at 

2:00 PM in Courtroom 27A. hi accordance with Rule 16.3(a) of the Local Civil 
Rules and Federal Rule of Civil Procedure 26(f)(1), counsel (including any 
nonprisoner pro se party) shall confer at least 21 days prior to the date of the 
Initial Scheduling Conference to discuss the matters outlined in Local 

CivilRule 16.3(c). Pursuant to Local Civil rule 16.3(d) and Federal Rule of 

Civil Procedure 26(f)(2), coimsel shall submit to the Court no later than 14 
days following then - meeting i.e., no fewer than 7 days prior to the date of the 
Initial Scheduling Conference a joint Report that, among otherthings: (1) 
outlines a detailed discovery plan as described in Federal Rule of Civil 

Procedure 26(f)(3); (2) addresses all topics listed in Local Civil rule 16.3(c); 
and (3) sets forth a proposed scheduling order. See LCvR 16.3(d). Coimsel are 
also directed to include in then - Report a brief statement of the case and the 
statutory basis for all causes of action and defenses. Parties are to communicate 
with the Court by motion, opposition, reply, or notice, not byletter. All 
inquiries concerning the status or scheduling of any pending matter shall be 
directed to the Courtroom Deputy Clerk, Ms. Terri Robinson, (202) 354-3179, 
rather than to chambers. If Ms. Robinson is unavailable, such inquiries shall be 
directed to the staff person in the Clerks Office who has been designated as her 
substitute. Chambers personnel will not handle questions relating to the status 
or scheduling of pending matters, nor will chambers provide legal advice of 
any kind. In an emergency, however, chambers can be contacted at (202) 354- 
3480.(SEE ORDER FOR FULL DETAILS). Signed by Judge Christopher R. 
Cooper on 9/18/2014. (ter) (Entered: 09/18/2014) 

10/06/2014 

— 

MOTION for Leave to Appear Pro Hac Vice :Attomey Name- Corynne 
McSherry, :Firm- Electronic Frontier Foundation,: Address- 815 Eddy Street, 
San Francisco, CA 94109. Phone No. - (415) 436-9333. Fax No. - (415) 436- 
9993 Filing fee $ 100, receipt number 0090-3862145. Fee Status: Fee Paid, by 
PUBLIC.RESOURCE.ORG, INC. (Attachments: # \ Affidavit of Corynne 
McSherry, # 2 Text of Proposed Order)(Stoltz, Mitchell) (Entered: 10/06/2014) 

10/06/2014 
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MINUTE ORDER granting 19 Motion for Leave to Appear Pro Hac Vice. 
Signed by Judge Christopher R. Cooper on 10/6/2014. (Iccrc2,) (Entered: 
10/06/2014) 

10/09/2014 

20 

MEET AND CONFER STATEMENT. (Hudis, Jonathan) (Entered: 

10/09/2014) 

10/09/2014 

21 

NOTICE offiling of Proposed Order Regarding Confidentiality of Discovery 
Material and Inadvertent Disclosure ofPrivileged Material filed by 
AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC., 
AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL 
COUNCIL ON MEASUREMENT IN EDUCATION, INC. (Hudis, Jonathan) 
(Entered: 10/09/2014) 

10/16/2014 


Minute Entry for proceedings held before Judge Christopher R. Cooper: Initial 
Scheduling Conference held and concluded on 10/16/2014. Comt to issue 
Scheduling Order accordingly. (Court Reporter Barbara DeVico) (ter) 

(Entered: 10/16/2014) 

10/16/2014 

22 

SCHEDULING ORDER: Upon consideration of the parties' Joint Meet and 
Confer Statement 20 , it is hereby ORDERED that the parties shall abide by the 
schedule in the attached Order. The deadline for the close of all discovery is 

July 13, 2015. A Post Discovery Status Conference is set for July 15, 2015 at 
2:00 PM in Courtroom 27A before Judge Christopher R. Cooper. Signed by 
Judge Christopher R. Cooper on 10/16/2014. (Iccrc2, ) (Entered: 10/16/2014) 

10/17/2014 

23 

Case reassigned to Judge Tanya S. Chutkan as related to CA 13-1215. Judge 
Christopher R. Cooper no longer assigned to the case, (ztnr, ) (Entered: 
10/20/2014) 

10/29/2014 

24 

ORDER: Entering the joint 2J_ Protective Order submitted by the parties. 

Signed by Judge Tanya S. Chutkan on 10/29/2014. (Ictsc2) (Entered: 

10/29/2014) 

11/21/2014 


MINUTE ORDER: Setting Hearing on .15 MOTION to Strike Defendant's Jury 
Demand. Motion Hearing set for 12/4/2014 11:30 AM in Courtroom 2 before 
Judge Tanya S. Chutkan. Signed by Judge Tanya S. Chutkan on 11/21/2014. 
(Ictsc2) (Entered: 11/21/2014) 

12/04/2014 


Minute Entry for proceedings held before Judge Tanya S. Chutkan: Motion 
Hearing held on 12/4/2014 re J_5 MOTION to Strike Defendant's Jury Demand 
filed by AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC., 
AMERICAN PSYCHOLOGICAL ASSOCIATION, INC. and NATIONAL 
COUNCIL ON MEASUREMENT IN EDUCATION, INC. Oral arugment 
heard, and motion taken under advisement.(Court Reporter: Janice Dickman.) 

(tj) (Entered: 12/04/2014) 

12/11/2014 

25 

MOTION to Compel filed by AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, 
INC., NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION, INC. 
(Attachments: # J. Declaration of Jonathan Hudis, # 2 Exhibit A to Hudis Decl, 

# 3 Exhibit B to Hudis Decl, # 4 Exhibit C to Hudis Decl, # 5 Exhibit D to 

Hudis Decl, # 6 Exhibit E to Hudis Decl, # 7 Exhibit F to Hudis Decl, # 8 


JA2133 

https://ecf.dcd.uscomts.gov/cgi-bin/DktRpt.pl? 118975691614536-L1 0-1 















District of Columbia live database Page 12 of 35 


USCA Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 375 of 573 




Exhibit G to Hudis Decl, # 9 Exhibit H to Hudis Decl, # K) Exhibit I to Hudis 
Decl, # LI Exhibit J to Hudis Decl, # 12 Exhibit K to Hudis Decl, # 13 Exhibit 

L to Hudis Decl, # 14 Exhibit M to Hudis Decl, # 15 Exhibit N to Hudis Decl, 

# H5 Exhibit O to Hudis Decl, # 17 Exhibit P to Hudis Decl, # J _8 Exhibit Q to 
Hudis Decl, # 19 Exhibit R to Hudis Decl, # 20 Text of Proposed Order) 

(Hudis, Jonathan) (Entered: 12/11/2014) 

12/12/2014 


MINUTE ORDER: Denying without prejudice 25 Plaintiffs' Motion to Compel 
for failing to comply with paragraph 3 of the Court's Scheduling Order 22 
requiring a notice to the Court and a joint telephone call to chambers over 
discovery disputes. Signed by Judge Tanya S. Chutkan on 12/12/14. (DJS) 
(Entered: 12/12/2014) 

12/12/2014 


MINUTE ORDER OF REFERRAL: The Court has determined that this action 
should be referred to Magistrate Judge Deborah A. Robinson for all issues 
related to DISCOVERY. The Court finds that because a similar case with the 
same defendant has already been referred to Judge Robinson for discovery 
purposes, the interests of judicial efficiency and economy will be served by 
Judge Robinson handling both matters. The parties are reminded, pursuant to 
LCvR 73.1, that this action may be referred for all purposes, including trial, 
upon the filing of an executed notice of consent by all parties. Consent of the 
District Court Judge is not necessary. Accordingly, it is hereby ORDERED that 
this action is referred to Magistrate Judge Deborah A. Robinson for discovery 
only, beginning immediately; and it is FURTHER ORDERED that any future 
filings related to discovery in this action shall have the initials of Judge Tanya 
Chutkan and Magistrate Judge Deborah A. Robinson following the case 
number in the caption. Signed by Judge Tanya S. Chutkan on 12/12/14. (DJS) 
(Entered: 12/12/2014) 

12/12/2014 

26 

CASE REFERRED to Magistrate Judge Deborah A. Robinson for Discovery, 
(md,) (Entered: 12/15/2014) 

12/15/2014 

21 

Amended MOTION to Compel discovery, privilege log, and further initial 
disclosures filed by AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, 
INC., NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION, INC. 
(Attachments: # 1 Declaration of Jonathan Hudis, # 2 Exhibit A to Hudis 
Declaration, # 3 Exhibit B to Hudis Declaration, # 4 Exhibit C to Hudis 
Declaration, # 5 Exhibit D to Hudis Declaration, # 6 Exhibit E to Hudis 
Declaration, # 7 Exhibit F to Hudis Declaration, # 8 Exhibit G to Hudis 
Declaration, # 9 Exhibit H to Hudis Declaration, # J_0 Exhibit I to Hudis 
Declaration, # JT Exhibit J to Hudis Declaration, # T2 Exhibit K to Hudis 
Declaration, # J_3 Exhibit L to Hudis Declaration, # J_4 Exhibit M to Hudis 
Declaration, # J_5 Exhibit N to Hudis Declaration, # _16 Exhibit O to Hudis 
Declaration, # L7 Exhibit P to Hudis Declaration, # J _8 Exhibit Q to Hudis 
Declaration, # Jj) Exhibit R to Hudis Declaration, # 20 Text of Proposed 
Order)(Hudis, Jonathan) Modified on 12/16/2014 (td, ). (Entered: 12/15/2014) 

12/16/2014 


Set/Reset Hearings: Motion Hearing on Plaintiffs' Amended Motion to Compel 
Discovery, Privilege Log, and Further Initial Disclosures (Document No. 27) is 
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scheduled for 11:00 AM on Thursday, 1/15/2015 in Courtroom 4 before 
Magistrate Judge Deborah A. Robinson. (SRH) (Entered: 12/16/2014) 

12/24/2014 

28 

Consent MOTION for Extension of Time to Oppose Plaintiffs' Motion to 
Compel by PUBLIC.RESOURCE.ORG, INC. (Attachments: #1 Text of 
Proposed Order)(Bridges, Andrew) (Entered: 12/24/2014) 

12/31/2014 


MINUTE ORDER: It is hereby ORDERED that Defendant/Counterclaimant's 
Consent Motion for Extension of Time to Oppose Plaintiffs/Coimterdefendants' 
Amended Motion to Compel Discovery, Privilege Log, and Further Initial 
Disclosures (Document No. 28) is GRANTED. It is FURTHER ORDERED 
that Defendant-counterclaimant shall file its opposition to Plaintiffs' Amended 
Motion to Compel Discovery, Privilege Log, and Further Initial Disclosures 
(Document No. 27) by no later than 6 p.m. on January 5, 2015. Signed by 
Magistrate Judge Deborah A. Robinson on December 31, 2014. (SRH) 

(Entered: 12/31/2014) 

01/05/2015 

29 

Memorandum in opposition to re 27 Amended MOTION to Compel filed filed 
by PUBLIC.RESOURCE.ORG, INC.. (Attachments: # \ Declaration of 

Andrew P. Bridges in Support of Defendant-Counterclaimant 
Public.Resource.Org, Inc.'s Opposition to Plaintiffs-Coimterdefendants' 
Amended Motion to Compel, # 2 Exhibit 1 to Declaration of Andrew P. 

Bridges, # 3 Exhibit 2 to Declaration of Andrew P. Bridges, # 4 Exhibit 3 to 
Declaration of Andrew P. Bridges, # 5 Exhibit 4 to Declaration of Andrew P. 
Bridges, # 6 Exhibit 5 to Declaration of Andrew P. Bridges, # 7 Exhibit 6 to 
Declaration of Andrew P. Bridges, # 8 Exhibit 7 to Declaration of Andrew P. 
Bridges)(Bridges, Andrew) (Entered: 01/05/2015) 

01/12/2015 

30 

REPLY to opposition to motion re 27 Amended MOTION to Compel filed on 
December 15, 2014, filed by AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, 
INC., NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION, 

INC.. (Attachments: # 1 Declaration in Reply of Jonathan Hudis, # 2 Exhibit S 
to Hudis Reply Decl, # 3 Exhibit T to Hudis Reply Decl, # 4 Exhibit U to 

Hudis Reply Decl, # 5 Exhibit V to Hudis Reply Decl, # 6 Exhibit W to Hudis 
Reply Decl, # 7 Exhibit X to Hudis Reply Decl, # 8 Exhibit Y to Hudis Reply 
Decl, # 9 Exhibit Z to Hudis Reply Decl, # K) Exhibit AA to Hudis Reply 

Decl, # JT Exhibit BB to Hudis Reply Decl, # 12 Exhibit CC to Hudis Reply 
Decl, # 13 Exhibit DD to Hudis Reply Decl, # 14 Exhibit EE to Hudis Reply 
Decl, # 15 Text of Proposed Order -Revised)(Hudis, Jonathan) (Entered: 
01/12/2015) 

01/12/2015 

3! 

MOTION for Leave to Appear Pro Hac Vice : Attorney Name- Matthew B. 
Becker, :Firm- Fenwick & West LLP, :Address- 801 California Street, 

Mountain View, CA 94041. Phone No. - (650) 335-7930. Fax No. - (650) 938- 
5200 Motion to Admit Matthew B. Becker Pro Hac Vice Filing fee $ 100, 
receipt number 0090-3960268. Fee Status: Fee Paid, by 
PUBLIC.RESOURCE.ORG, INC. (Attachments: # 1 Declaration, # 2 Text of 
Proposed Order)(Bridges, Andrew) (Entered: 01/12/2015) 

01/15/2015 


Set/Reset Hearings: Motion Hearing on Plaintiffs' Amended Motion to Compel 
Discovery, Privilege Log, and Further Initial Disclosures (Document No. 27) is 
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rescheduled to 1/22/2015 09:30 AM in Courtroom 4 before Magistrate Judge 
Deborah A. Robinson, (zcmin, ) (Entered: 01/15/2015) 

01/16/2015 

32 

NOTICE of Change of Address by Jonathan Hudis (Hudis, Jonathan) (Entered: 
01/16/2015) 

01/22/2015 


MINUTE ORDER: It is hereby ORDERED that Defendant's 31 Motion for 
Leave to Appear Pro Hac Vice is GRANTED. Matthew B. Becker is hereby 
admitted pro hac vice to appear and participate as co-counsel in the above- 
referenced action. Signed by Magistrate Judge Deborah A. Robinson on 

1/22/15. (zcinm,) (Entered: 01/22/2015) 

01/22/2015 


Minute Entry for proceedings held before Magistrate Judge Deborah A. 
Robinson: Motion Healing held on 1/22/2015 re 27 Amended MOTION to 
Compel filed by AMERICAN EDUCATIONAL RESEARCH 

ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, 
INC., NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION, INC. 
Defendant shall file any motion to consolidate this and the related case (no. 13- 
cv-01215) for discovery pmposes by no later than 1/29/2015. The court will 
take the pending motion 27 Plaintiffs' Amended Motion to Compel under 
advisement. (Court Reporter Bowles Reporting Service)(FTR Time Frame: 
9:47:06 - 10:33:23 - 10:42:48 - 11:38:18, Crtrm4) (zcrnrn, ) (Entered: 
01/22/2015) 

01/29/2015 

33 

WITHDRAWN PURSUANT TO ENTRY NO.: 41 .MOTION to 

Consolidate Cases Defendant-Counterclaim Public.Resource.Org, Inc.'s 

Motion to Consolidate for the Purposes of Discovery by 
PUBLIC.RESOURCE.ORG, INC. (Attachments: # \ Text of Proposed Order 
[Proposed] Order Granting Defendant's Motion to Consolidate for Purposes of 
Discovery (Dkt. No. 33))(Bridges, Andrew) Modified on 3/2/2015 (td,). 
(Entered: 01/29/2015) 

01/29/2015 

34 

AFFIDAVIT re 33 MOTION to Consolidate Cases Defendant-Counterclaim 
Public.Resource. Org, Inc. 's Motion to Consolidate for the Purposes of 
Discovery Declaration of Mattlmv B. Becker in Support of Defendant- 
Counterclaim Public.Resource.Org, Inc. 's Motion to Consolidate for the 
Purposes of Discovery (Dkt. 33) by PUBLIC.RESOURCE.ORG, INC.. 
(Attachments: # I Exhibit l)(Bridges, Andrew) (Entered: 01/29/2015) 

02/02/2015 

35 

MEMORANDUM AND OPINION. Signed by Judge Tanya S. Chutkan on 
2/2/2015. (Ictsc2) (Entered: 02/02/2015) 

02/02/2015 

36 

ORDER granting 15 Motion to Strike. The jury demand in Defendant's 12 
counterclaim and Answer is stricken. Signed by Judge Tanya S. Chutkan on 
2/2/2015. (Ictsc2) (Entered: 02/02/2015) 

02/08/2015 

37 

TRANSCRIPT OF PROCEEDINGS before Magistrate Judge Deborah A. 
Robinson held on 1/22/2015; Page Numbers: 1-73. Court Reporter/Transcriber 
Bowles Reporting Seivice, Telephone number 860-464-1083, Court Reporter 
Email Address : brs-ct@sbcglobal.net. 

For the first 90 days after this filing date, the transcript may be viewed at the 
courthouse at a public terminal or purchased from the court reporter referenced 
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above. After 90 days, the transcript may be accessed via PACER. Other 
transcript formats, (multi-page, condensed, CD or ASCII) may be purchased 
from the court reporter. 

NOTICE RE REDACTION OF TRANSCRIPTS: The parties have twenty- 
one days to file with the court and the court reporter any request to redact 
personal identifiers from this transcript. If no such requests are filed, the 
transcript will be made available to the public via PACER without redaction 
after 90 days. The policy, which includes the five personal identifiers 
specifically covered, is located on our website at www.dcd.uscourts.gov. 

Redaction Request due 3/1/2015. Redacted Transcript Deadline set for 

3/11/2015. Release of Transcript Restriction set for 5/9/2015.(znrnw, ) 

(Entered: 02/09/2015) 

02/10/2015 

38 

Consent MOTION to Intervene by AMERICAN SOCIETY FOR TESTING 
AND MATERIALS, NATIONAL FIRE PROTECTION ASSOCIATION, 

INC., AMERICAN SOCIETY OF HEATING, REFRIGERATING, AND 
AIR-CONDITIONING ENGINEERS, INC. (Attachments: # 1 Text of 

Proposed Oider)(Fee, J.) (Entered: 02/10/2015) 

02/11/2015 


MINUTE ORDER: Granting consent 38 Motion to Intervene for the limited 
purpose of opposing Defendant's 33 Motion to Consolidate for the purposes of 
Discovery. Oppositions to the Motion to Consolidate are due by February 17, 
2015; reply due by February 27, 2015. Signed by Judge Tanya S. Chutkan on 

2/11/2015. (Ictsc2) (Entered: 02/11/2015) 

02/12/2015 


Set/Reset Deadlines: Opposition due by 2/17/2005. Reply due by 2/27/2015. 
(sm) (Entered: 02/12/2015) 

02/17/2015 

39 

Memorandum in opposition to re 33 MOTION to Consolidate Cases 
Defendant-Counterclaim Public.Resource.Org, Inc.'s Motion to Consolidate 
for the Purposes of Discovery filed by AMERICAN SOCIETY FOR 

TESTING AND MATERIALS, AMERICAN SOCIETY OF HEATING, 
REFRIGERATING, AND AIR-CONDITIONING ENGINEERS, INC., 
NATIONAL FIRE PROTECTION ASSOCIATION, INC.. (Attachments: # \ 
Exhibit A, # 2 Exhibit B, # 3 Exhibit C, # 4 Exhibit D)(Fee, J.) (Entered: 
02/17/2015) 

02/18/2015 

40 

Memorandum in opposition to re 33 MOTION to Consolidate Cases 
Defendant-Counterclaim Public.Resource.Org, Inc.'s Motion to Consolidate 
for the Purposes of Discovery filed by AMERICAN EDUCATIONAL 
RESEARCH ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL 
ASSOCIATION, INC., NATIONAL COUNCIL ON MEASUREMENT IN 
EDUCATION, INC.. (Attachments: # l Declaration of Jonathan Hudis, # 2 
Exhibit A to Hudis Declaration, # 3 Exhibit B to Hudis Declaration, # 4 Exhibit 
C to Hudis Declaration, # 5 Exhibit D to Hudis Declaration, # 6 Exhibit E to 
Hudis Declaration, # 7 Text of Proposed Order)(Hudis, Jonathan) (Entered: 
02/18/2015) 

02/26/2015 

41 

WITHDRAWAL of Motion by PUBLIC.RESOURCE.ORG, INC. re 33 
MOTION to Consolidate Cases Defendant-Counterclaim Public.Resource.Org, 
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Inc. 's Motion to Consolidate for the Purposes of Discovery filed by 
PUBLIC.RESOURCE.ORG, INC.. (Bridges, Andrew) (Entered: 02/26/2015) 

03/04/2015 

42 

Consent MOTION for Extension of Time to Complete Discovery filed by 
AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC., 
AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL 
COUNCIL ON MEASUREMENT IN EDUCATION, INC. (Attachments: # 1 
Text of Proposed Order)(Hudis, Jonathan) (Entered: 03/04/2015) 

03/09/2015 


Set/Reset Hearings: Status Conference, including consideration of 42 
scheduled for 3/19/2015 at 02:00 PM in Courtroom 4 before Magistrate Judge 
Deborah A. Robinson. All coimsel shall meet and confer in advance of said 
hearing in an effort to reach a consensus regarding the expeditious completion 
of discovery. (Icdar2) (Entered: 03/09/2015) 

03/17/2015 

43 

MEET AND CONFER STATEMENT. (Hudis, Jonathan) (Entered: 

03/17/2015) 

03/19/2015 


Minute Entry for proceedings held before Magistrate Judge Deborah A. 
Robinson: Status Conference held on 3/19/2015. (Court Reporter Lisa Moreira) 
(zcmm,) (Entered: 03/19/2015) 

03/23/2015 


Minute Entry for proceedings held before Magistrate Judge Deborah A. 
Robinson: Status Conference conducted on 3/19/2015; pending motions 
Document No. 27 and Document No. 42 taken under advisement. (FTR Time 
Frame: 2.5) (lcdarl, ) (Entered: 03/23/2015) 

03/26/2015 

44 

TRANSCRIPT OF PROCEEDINGS before Magistrate Judge Deborah A. 
Robinson held on March 19, 2015; Page Numbers: 1-60. Date of 

Issuance:March 26, 2015. Court Reporter/Transcriber Lisa A. Moreira, RDR, 
CRR, Telephone number 202-354-3187, Court Reporter Email Address : 
Lisa_Moreira@dcd.uscourts. gov. 

For the first 90 days after this filing date, the transcript may be viewed at the 
courthouse at a public terminal or purchased from the court reporter referenced 
above. After 90 days, the transcript may be accessed via PACER. Other 
transcript formats, (multi-page, condensed, CD or ASCII) may be purchased 
from the court reporter. 

NOTICE RE REDACTION OF TRANSCRIPTS: The parties have twenty- 
one days to file with the court and the court reporter any request to redact 
personal identifiers from this transcript. If no such requests ar e filed, the 
transcript will be made available to the public via PACER without redaction 
after 90 days. The policy, which includes the five personal identifiers 
specifically covered, is located on our website at www.dcd.uscom1s.gov. 

Redaction Request due 4/16/2015. Redacted Transcript Deadline set for 
4/26/2015. Release of Transcript Restriction set for 6/24/2015.(Moreira, Lisa) 
(Entered: 03/26/2015) 

04/09/2015 

45 

MOTION for Hearing re 27 Amended MOTION to Compel filed, 42 Consent 
MOTION for Extension of Time to Complete Discovery filed without 


JA2138 

https://ecf.dcd.uscomts.gov/cgi-bin/DktRpt.pl? 118975691614536-L1 0-1 













District of Columbia live database 


Page 17 of 35 

Filed: 01/31/2018 Page 380 of 573 


USCA Case #17-7035 Document #1715850 




opposition from Defendant by AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, 
INC., NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION, INC. 
(Hudis, Jonathan) (Entered: 04/09/2015) 

04/15/2015 

46 

STIPULATION re 27 Amended MOTION to Compel filed, 42 Consent 
MOTION for Extension of Time to Complete Discovery filed, 45 MOTION for 
Healing re 27 Amended MOTION to Compel filed , 42 Consent MOTION for 
Extension of Time to Complete Discovery filed without opposition from 
Defendant — submitted bv Public.Resource.Org, Inc. and by AMERICAN 
EDUCATIONAL RESEARCH ASSOCIATION, INC., AMERICAN 
PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC.. (Hudis, Jonathan) (Entered: 
04/15/2015) 

04/21/2015 


MINUTE ORDER: Motion for Hearing 45 is hereby GRANTED; a Status 
Healing is scheduled for 10:00 AM on Wednesday May 6, 2015, counsel shall 
appear in courtroom 4. Signed by Magistrate Judge Deborah A. Robinson on 
4/21/2015. (lcdarl,) (Entered: 04/21/2015) 

04/21/2015 


MINUTE ENTRY: Status Hearing re 45 set for 5/6/2015 10:00 AM in 
Courtroom 4 before Magistrate Judge Deborah A. Robinson. Counsel shall 
appeal’. Signed by Magistrate Judge Deborah A. Robinson on 4/21/2015. 

(lcdarl,) (Entered: 04/21/2015) 

05/05/2015 


MINUTE ORDER: Motion for Extension of Time to Complete Discovery 42 is 
hereby GRANTED. Signed by Magistrate Judge Deborah A. Robinson on 

5/5/2015.(lcdarl, ) (Entered: 05/05/2015) 

05/05/2015 


Status Hearing: Status Hearing re 45 set for 5/6/2015 at 10:00 AM has been 
CONTINUED to 5/13/2015 at 2:00 PM in Courtroom 4 before Magistrate 

Judge Deborah A. Robinson. Counsel shall appear. The Court apologizes for 
any inconvenience this may cause the parties. Signed by Magistrate Judge 
Deborah A. Robinson on 4/21/2015. (lcdarl, ) (Entered: 04/21/2015) (Entered: 
05/05/2015) 

05/08/2015 


Set/Reset Hearings: As a result of a scheduling conflict, it is necessaiy for the 
court to again continue the status conference. Status Conference now scheduled 
for 05/13/2015 is hereby CONTINUED to 2:00 PM on 5/21/2015 in 

Courtroom 4 before Magistrate Judge Deborah A. Robinson. The court again 
apologizes to coimsel and parties for any inconvenience, (lcdarl, ) (Entered: 
05/08/2015) 

05/18/2015 

47 

MOTION for Extension of Time to Complete Discovery filed by AMERICAN 
EDUCATIONAL RESEARCH ASSOCIATION, INC., AMERICAN 
PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC. (Attachments: # \ Text of 

Proposed Order)(Hudis, Jonathan) (Entered: 05/18/2015) 

05/19/2015 

48 

NOTICE Consent Request for Telephonic Status Conference [Dkt. No. 45] by 
PUBLIC.RESOURCE.ORG, INC. re Set/Reset Hearings, (Becker, Matthew) 
(Entered: 05/19/2015) 
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05/20/2015 

49 

ORDER granting in part and denying in part Motion to Compel Document No. 
27 . Signed by Magistrate Judge Deborah A. Robinson on 05/20/2015.(lcdarl,) 
(Entered: 05/20/2015) 

05/20/2015 


Set/Reset Hearings: Status Conference is hereby set for 6/11/2015 at 02:00 PM 
in Courtroom 4 before Magistrate Judge Deborah A. Robinson, (lcdarl, ) 
(Entered: 05/20/2015) 

05/21/2015 


MINUTE ORDER: Consent Request for Telephonic Status Conference 48 is 
hereby DENIED as moot. Signed by Magistrate Judge Deborah A. Robinson 
on 5/21/2015.(lcdarl,) (Entered: 05/21/2015) 

06/03/2015 

50 

NOTICE of Change of Address by Jonathan Hudis (Hudis, Jonathan) (Entered: 
06/03/2015) 

06/04/2015 

51 

Memorandum in opposition to re 47 MOTION for Extension of Time to 
Complete Discovery filed Public Resource's Opposition to Plaintiffs' Motion to 
Extend Time for Fact Discovery filed by PUBLIC.RESOURCE.ORG, INC.. 
(Bridges, Andrew) (Entered: 06/04/2015) 

06/08/2015 

52 

Consent MOTION for Order Request for Telephonic Status Conference by 
PUBLIC.RESOURCE.ORG, INC. (Becker, Matthew) (Entered: 06/08/2015) 

06/08/2015 


MINUTE ORDER granting 52 Consent Motion for Telephonic Status 
Conference. Signed by Magistrate Judge Deborah A. Robinson on June 8, 

2015. (Icdar2) (Entered: 06/08/2015) 

06/11/2015 


Minute Entry for Telephone Status Conference held before Magistrate Judge 
Deborah A. Robinson on 6/11/15 : For the reasons stated on the record. 
Plaintiffs' Motion to Extend Time for Fact Discovery 47 is denied. The parties 
shall comply with the dates previously set, which seive to modify the 

Scheduling Order of 10/16/14. Court Repoiter: USDC Court Reporters - Ctnn. 

4; FTR Time Frame: 3:01:56 - 3:27:13. Plaintiffs' Counsel: Jonathan Hudis, 
Kathleen Cooney-Porter and Kate Cappaert; Defendant's Counsel: Matthew 
Becker, David Halperin and Mitchell Stoltz. (kk) (Entered: 06/12/2015) 

06/11/2015 

53 

ORDER ON CONSENT MOTION TO EXTEND TIME FOR DISCOVERY 
AND CASE SCHEDULE: Fact Discovery closes on 5/18/15; Opening Expert 
Disclosures due 6/15/15; Rebuttal Expert Disclosures due 7/15/15; Replies to 
Rebuttal Disclosures due 7/29/15; Final Replies to Expert Disclosures due 
8/12/15; Discovery closes on 9/11/15; Post-Discovery Conference set for 

9/15/15 at 2:00 PM in Courtroom 4 before Magistrate Judge Deborah A. 
Robinson; signed by Magistrate Judge Deborah A. Robinson on 6/11/15, nunc 
pro tunc to 5/5/15. (kk) (Entered: 06/12/2015) 

07/06/2015 

54 

TRANSCRIPT OF TELEPHONE STATUS CONFERENCE before Magistrate 
Judge Deborah A. Robinson held on 6-11-2015; Page Numbers: 1-19. Date of 
Issuance:7-6-2015. Transcriber Annette M. Montalvo, Telephone number 202- 
354-3111, Transcriber/Court Reporter Email Address : 
annette.montalvo@gmail.com. 

For the first 90 days after this filing date, the transcript may be viewed at the 
courthouse at a public terminal or purchased from the court reporter referenced 
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above. After 90 days, the transcript may be accessed via PACER. Other 
transcript formats, (multi-page, condensed, CD or ASCII) may be purchased 
from the court reporter. 

NOTICE RE REDACTION OF TRANSCRIPTS: The parties have twenty- 
one days to file with the court and the court reporter any request to redact 
personal identifiers from this transcript. If no such requests are filed, the 
transcript will be made available to the public via PACER without redaction 
after 90 days. The policy, which includes the five personal identifiers 
specifically covered, is located on our website at www.dcd.uscourts.gov. 

Redaction Request due 7/27/2015. Redacted Transcript Deadline set for 
8/6/2015. Release of Transcript Restriction set for 10/4/2015.(Montalvo, 
Annette) Modified on 7/8/2015 (zrdj). (Entered: 07/06/2015) 

07/09/2015 


MINUTE ORDER: The status conference previously set for 7/15/15 is hereby 
vacated. All deadlines set forth in Judge Robinson's 06/11/2015 Scheduling 
Order 53 shall remain in effect. Signed by Judge Tanya S. Chutkan on 7/9/15. 
(DJS) (Entered: 07/09/2015) 

09/09/2015 

55 

STIPULATION re Order JOINT STIPULATION AND [PROPOSED] ORDER 
EXTENDING COMPLETION DATE OF DEPOSITION OF PLAINTIFFS' 
EXPERT DR. PHILLIPS TO SEPTEMBER 22, 2015 by 
PUBLIC.RESOURCE.ORG, INC.. (Becker, Matthew) (Entered: 09/09/2015) 

09/10/2015 


MINUTE ORDER: The court has reviewed the Joint Stipulation And Proposed 
Order To Extend The Date On Which Expert Deposition May Be Taken 
(Document No. 55 ) filed by the parties on September 9, 2015. Counsel for the 
parties are hereby reminded that the applicable local and federal rules require 
that requests for action by the court be made by motion. Accordingly, if it 
remains the intention of the parties to request an extension of any deadline, 
counsel shall do so by motion. Signed by Magistrate Judge Deborah A. 

Robinson on 9/10/2015. (lcdarl, ) (Entered: 09/10/2015) 

09/10/2015 

56 

Joint MOTION for Extension of Time to Extend the Date on Which Expert 
Deposition Mav Be Taken and Proposed Order by 

PUBLIC.RESOURCE.ORG, INC. (Becker, Matthew) (Entered: 09/10/2015) 

09/10/2015 

57 

Consent MOTION for Order Request for Telephonic Status Conference by 
PUBLIC.RESOURCE.ORG, INC. (Becker, Matthew) (Entered: 09/10/2015) 

09/11/2015 


MINUTE ORDER: Motion for Extension of Time 56 and Consent Motion for 
Telephonic Status Conference 57 are hereby GRANTED. It is further 
ORDERED that the Status Conference now scheduled for September 15, 2015, 
is hereby CONTINUED to September 29, 2015 at 4:00 PM. Signed by 
Magistrate Judge Deborah A. Robinson on 9/11/2015. (lcdarl, ) (Entered: 
09/11/2015) 

09/11/2015 


Set/Reset Healings: Status Conference is hereby set for 9/29/2015 at 04:00 PM 
in Courtroom 4 before Magistrate Judge Deborah A. Robinson, (lcdarl, ) 
(Entered: 09/11/2015) 

09/29/2015 
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Minute Entry for proceedings held before Magistrate Judge Deborah A. 
Robinson: Post-Discovery Status Conference conducted by telephone on 
9/29/2015. All parties agree that Discoveiy is closed and that there are no 
disputes. Plaintiffs' Counsel: Kathleen Cooney-Porter and Jonathan Hudis; 
Defendant's Counsel: Matthew Becker and David Halperin. Court Reporter 

FTR Gold - Ctrm. 4. (FTR Time Frame: 4:11:00-4:16:10). (mr) (Entered: 
09/29/2015) 

10/27/2015 


MINUTE ORDER. A status conference will be held in both this case and 
American Society for Testing and Materials v. Public.Resource.Org, Inc., Civil 
Action No. l:13-cv-01215-TSC on Wednesday, November 4, 2015 at 10:15am. 
The court intends to set schedules for briefing summary judgment motions in 
both cases at the status conference. The paxties to this case are hereby directed 
to jointly file then proposed schedules for summary judgment briefing, 
accompanied by proposed orders, by Friday, October 30, 2015. Signed by 

Judge Tanya S. Chutkan on 10/27/15. (Ictsc2) (Entered: 10/27/2015) 

10/28/2015 


Set/Reset Deadlines/Hearings: Proposed Briefing Schedule due by 10/30/2015. 
Scheduling Conference set for 11/4/2015 at 10:15 AM in Courtroom 2 before 
Judge Tanya S. Chutkan. (zsm) (Entered: 10/28/2015) 

10/30/2015 

58 

PROPOSED BRIEFING SCHEDULE and Joint Report of the Parties, 
submitted by AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, 
INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL 
COUNCIL ON MEASUREMENT IN EDUCATION, INC.. (Attachments: # \ 
Exhibit A - Plaintiffs' Proposed Order, # 2 Exhibit B - Defendant's Proposed 
Order)(Hudis, Jonathan) (Entered: 10/30/2015) 

11/04/2015 


Minute Entry for proceedings held before Judge Tanya S. Chutkan: Status 
Conference held on 11/4/2015. Older to issue. (Court Reporter Biyan Wayne.) 
(zsm) (Entered: 11/04/2015) 

11/04/2015 


MINUTE ORDER setting briefing schedule: Plaintiffs' Motion for Summary 
Judgment due by December 21, 2015; Defendant's Opposition to Plaintiffs' 
Motion for Summary Judgment and COMBINED Cross-Motion for Summary 
Judgment due by January 21, 2016; Plaintiffs' Reply in Support of their Motion 
for Summary Judgment and COMBINED Opposition to Defendant's Cross- 
Motion for Summary Judgment due by Febniaxy 18, 2016; Defendant's Reply 
in Support of its Cross-Motion for Summary Judgment due by March 3, 2016; 
Amicus briefs due by February 11, 2016. Signed by Judge Tanya S. Chutkan 
on 11/4/15. (Ictsc2) (Entered: 11/04/2015) 

11/04/2015 


Set/Reset Deadlines/Hearings: Summary Judgment motions due by 

12/21/2015. Response to Motion for Summary Judgment due by 1/21/2016. 
Reply to Motion for Summary Judgment due by 2/18/2016. Brief due by 

2/11/2016. Replies due by 3/3/2016. VACATED PURSUANT TO MINUTE 

ORDER FILED ON 11/5/2015.Motion Healing set for 3/22/2016 at 9:30 

AM in Courtroom 2 before Judge Tanya S. Chutkan. (zsm) Modified on 
11/6/2015 (ttli). (Entered: 11/04/2015) 

11/05/2015 


MINUTE ORDER: Due to an unexpected scheduling conflict, the motion 
hearing previously set for 3/22/2016 is hereby VACATED. A new date will be 
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set at a later time. Signed by Judge Tanya S. Ckutkan on 11/5/15. (DJS) 
(Entered: 11/05/2015) 

11/14/2015 

59 

TRANSCRIPT OF 11/04/15 STATUS HEARING before Judge Tanya S. 
Chutkan, held on November 4,2015. Page Numbers: 1-21. Date of Issuance: 
11/14/15. Court Reporter: Bryan A. Wayne; telephone number: 202-354-3186. 
Transcripts mav be ordered bv submittine the Transcript Order Form. 

For the first 90 days after this filing date, the transcript may be viewed at the 
courthouse at a public terminal or purchased from the court reporter referenced 
above. After 90 days, the transcript may be accessed via PACER. Other 
transcript formats, (multi-page, condensed, CD or ASCII) may be purchased 
from the court reporter. 

NOTICE RE REDACTION OF TRANSCRIPTS: The parties have twenty- 
one days to file with the court and the court reporter any request to redact 
personal identifiers from this transcript. If no such requests are filed, the 
transcript will be made available to the public via PACER without redaction 
after 90 days. The policy, which includes the five personal identifiers 
specifically covered, is located on our website at www.dcd.uscourts.gov. 

Redaction Request due 12/5/2015. Redacted Transcript Deadline set for 
12/15/2015. Release of Transcript Restriction set for 2/12/2016.(Wayne, 

Bryan) (Entered: 11/14/2015) 

12/21/2015 

60 

MOTION for Summary Judgment Filed by AMERICAN EDUCATIONAL 
RESEARCH ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL 
ASSOCIATION, INC., NATIONAL COUNCIL ON MEASUREMENT IN 
EDUCATION, INC. (Attachments: # I Statement of Facts Points of Authority, 

# 2 Statement of Facts Statement of Undisputed Facts, # 3 Declar ation 

Declar ation of Jonathan Hudis, # 4 Exhibit Ex. A, # 5 Exhibit Ex. B, # 6 

Exhibit Ex. C, # 7 Exhibit Ex. D, # 8 Exhibit Ex. E, # 9 Exhibit Ex. F. # 10 
Exhibit Ex. G, # U_ Exhibit Ex. H, # 12 Exhibit Ex. I, # 13 Exhibit Ex. J, # 14 
Exhibit Ex. K, # 15 Exhibit Ex. L, # 16 Exhibit Ex. M, # 17 Exhibit Ex. N, # 

18 Exhibit Ex. O, # 19 Exhibit Ex. P, # 20 Exhibit Ex. Q,#2\ Exhibit Ex. R, # 
22 Exhibit Ex. S, # 23 Exhibit Ex. T, # 24 Exhibit Ex. U, # 25 Exhibit Ex. V-l, 

# 26 Exhibit Ex. V-2, # 27 Exhibit Ex. W, # 28 Exhibit Ex. X, # 29 Exhibit Ex. 
Y, # 30 Exhibit Ex. Z,#3l Exhibit Ex. AA, # 32 Exhibit Ex. BB, # 33 Exhibit 
Ex. CC, # 34 Exhibit Ex. DD, # 35 Exhibit Ex. EE, # 36 Exhibit Ex. FF-1, # 37 
Exhibit Ex. FF-2, # 38 Exhibit Ex. FF-3, # 39 Exhibit Ex. FF-4, # 40 Exhibit 

Ex. FF-5, # 41 Exhibit Ex. FF-6, # 42 Exhibit Ex. GG, # 43 Exhibit Ex. HH, # 

44 Exhibit Ex. H, # 45 Exhibit Ex. JJ, # 46 Exhibit Ex. KK, # 47 Exhibit Ex. 

LL, # 48 Exhibit Ex. MM, # 49 Declaration Declaration of Marianne Ernesto, # 
50 Exhibit Ex. NN, # 51 Exhibit Ex. OO, # 52 Exhibit Ex. PP, # 53 Exhibit Ex. 
QQ, # 54 Exhibit Ex. RR, # 55 Exhibit Ex. SS, # 56 Exhibit Ex. TT, # 57 

Exhibit Ex. UU, # 58 Exhibit Ex. W, # 59 Exhibit Ex. WW, # 60 Exhibit Ex. 
XX, # 61 Exhibit Ex. YY, # 62 Exhibit Ex. ZZ, # 63 Exhibit Ex. AAA, # 64 
Exhibit Ex. BBB, # 65 Exhibit Ex. CCC, # 66 Exhibit Ex. DDD, # 67 Exhibit 
Ex. EEE, # 68 Exhibit Ex. FFF, # 69 Exhibit Ex. GGG, # 70 Exhibit Ex. HHH, 

# 21 Exhibit Ex. IH, # 72 Exhibit Ex. JJJ, # 73 Declaration Declaration of 
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Lauress Wise, # 74 Exhibit Ex. KKK, # 75 Exhibit Ex. LLL, # 76 Declaration 
Declar ation of Wayne Camara, # 77 Exhibit Ex. MMM, # 78 Declaration 
Declaration of Felice Levine, # 79 Exhibit Ex. NNN, # 80 Exhibit Ex. OOO 
(Public Version), # 81, Exhibit Ex. PPP, # 82 Exhibit Ex. QQQ, # 83 Exhibit 

Ex. RRR, # 84 Exhibit Ex. SSS, # 85 Exhibit Ex. TTT-1, # 86 Exhibit Ex. 
TTT-2, # 87 Exhibit Ex. UUU, # 88 Declaration Declaration of Kmt Geisinger, 

# 89 Declaration Declaration of Dianne Schneider, #90 Text of Proposed 

Order Proposed Order, # 91 Certificate of Service Certificate of Service) 

(Hudis, Jonathan). Added MOTION for Permanent Injunction on 12/22/2015 
(td). (Entered: 12/21/2015) 

12/21/2015 

61 

SEALED MOTION FOR LEAVE TO FILE DOCUMENT UNDER SEAL 
filed by AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC., 
AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL 
COUNCIL ON MEASUREMENT IN EDUCATION, INC. (This document is 
SEALED and only available to authorized persons.) (Attachments: # 2 Exhibit 
OOO - sought to be sealed, # 2 Memorandum in Support, # 3 Text of Proposed 
Order, # 4 Certificate of Service)(Hudis, Jonathan) (Entered: 12/21/2015) 

01/04/2016 


MINUTE ORDER: Granting 61. Plaintiffs Sealed Motion for Leave to File 
Document Under Seal. Exhibit OOO to the Declaration of Felice J. Levine in 
Support of Plaintiffs' Motion for Summary Judgment and a Permanent 

Injunction 60 shall be filed under seal. Exhibit OOO contains Plaintiffs' 
revenue and expenses associated with the preparation, publication, and 
advertising of the "Standards for Educational and Psychological Testing" (1999 
ed.)(the 1999 Standards). In this action. Plaintiffs assert that Defendant has 
infringed their copyright in the 1999 Standards by digitally copying the work 
and publishing it online. The court finds that there is a significant interest in 
preserving the confidentiality of Exhibit OOO and that there is no compelling 
interest for public disclosure of Exhibit OOO at this time. Signed by Judge 
Tanya S. Chutkan on 1/4/16. (Entered: 01/04/2016) 

01/04/2016 

62 

SEALED DOCUMENT (Exhibit OOO) filed by AMERICAN 

EDUCATIONAL RESEARCH ASSOCIATION, INC., AMERICAN 
PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC., re 60 MOTION for Summary 
Judgment Filed MOTION for Permanent Injunction filed by AMERICAN 
EDUCATIONAL RESEARCH ASSOCIATION, INC., AMERICAN 
PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC., Order on Sealed Motion for Leave 
to File Document Under Seal,,,. (This document is SEALED and only available 
to authorized persons.)(ztd) (Entered: 01/05/2016) 

01/13/2016 

63 

Consent MOTION for Leave to Appear - Pro Hac Vice : Attorney Nanre- 
Sebastian E. Kaplan, :Firm- Fenwick & West LLP, :Address- 555 California 
Street, 12th FI., San Francisco, CA 94104. Phone No. - (415) 875-2300. Fax 

No. - (415) 281-1350 Filing fee $ 100, receipt number 0090-4377619. Fee 

Status: Fee Paid, by PUBLIC.RESOURCE.ORG, INC. (Attachments: # j. 
Declaration of Sebastian Kaplan, # 2 Text of Proposed Order)(Stoltz, Mitchell) 
(Entered: 01/13/2016) 





JA2144 

https://ecf.dcd.uscomts.gov/cgi-bin/DktRpt.pl? 118975691614536-L1 0-1 











District of Columbia live database 


Page 23 of 35 

Filed: 01/31/2018 Page 386 of 573 


USCA Case #17-7035 Document #1715850 


01/19/2016 

64 

NOTICE of Appearance by Jonathan P. Labukas on behalf of AMERICAN 
EDUCATIONAL RESEARCH ASSOCIATION, INC., AMERICAN 
PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC. (Labukas, Jonathan) (Entered: 
01/19/2016) 

01/21/2016 

65 

NOTICE OF WITHDRAWAL OF APPEARANCE as to AMERICAN 
EDUCATIONAL RESEARCH ASSOCIATION, INC.. Attorney Kathleen 
Cooney-Porter terminated. (Cooney-Porter, Kathleen) (Entered: 01/21/2016) 

01/21/2016 

66 

SEALED MOTION FOR LEAVE TO FILE DOCUMENT UNDER SEAL 
filed by PUBLIC.RESOURCE.ORG, INC. (This document is SEALED and 
only available to authorized persons.) (Attachments: # J. Text of Proposed 

Order, # 2 Memorandum in Support of Defendant-Counterclaimant 
Public.Resource.Org's Motion to Strike, # 3 Declaration of Matthew Becker in 
Support of Motion to Strike, # 4 Exhibit 1, # 5 Exhibit 2, # 6 Exhibit 6, # 7 
Exhibit 7)(Bridges, Andrew) (Entered: 01/21/2016) 

01/21/2016 

67 

MOTION to Strike 60 the declaration of Kurt P. Geisinger by 
PUBLIC.RESOURCE.ORG, INC. (Attachments: # J_ Memorandum in Support 
of Defendant-Counterclaimant Public.Resource.Orgs Motion to Strike 
[PUBLIC], # 2 Declaration of Matthew Becker [PUBLIC], # 3 Exhibit 1 (Filed 
Under Seal), # 4 Exhibit 2 (Filed Under Seal), # 5 Exhibit 3, # 6 Exhibit 4, # 7 
Exhibit 5, # 8 Exhibit 6 (Filed Under Seal), # 9 Exhibit 7 (Filed Under Seal), # 

10 Exhibit 8, # J4 Exhibit 9, # 12 Exhibit 10, # 13 Exhibit 11,# JL4 Exhibit 12, 

# L5 Exhibit 13, # _16 Exhibit 14, # F7 Exhibit 15, # 18 Exhibit 16, # _19 Text of 
Proposed Order, # 20 Certificate of Service)(Bridges, Andrew) Modified on 
1/21/2016 linkage and text(td). (Entered: 01/21/2016) 

01/21/2016 

68 

SEALED MOTION FOR LEAVE TO FILE DOCUMENT UNDER SEAL 
filed by PUBLIC.RESOURCE.ORG, INC. (This document is SEALED and 
only available to authorized persons.) (Attachments: # l Text of Proposed 

Order Granting Public.Resource.Org's Motion to File Under Seal, # 2 
[SEALED] Memorandum of Points and Authorities In Support of Motion for 
Summary Judgment and Opposition to Plaintiffs' Motion for Summary 

Judgment and Permanent Injunction, # 3 [SEALED] Statement of Material 

Facts, # 4 [SEALED] Statement of Disputed Facts, # 5 [SEALED] Objections 
to Plaintiffs' Evidence, # 6 [Sealed] Exhibit 2, # 7 [Sealed] Exhibit 3, # 8 
[Sealed] Exhibit 4, # 9 [Sealed] Exhibit 5, # 10 [Sealed] Exhibit 6, # 11 
[Sealed] Exhibit 8, # 12 [Sealed] Exhibit 11, # 13 [Sealed] Exhibit 12, # 14 
[Sealed] Exhibit 13, # F5 [Sealed] Exhibit 14, # 16 [Sealed] Exhibit 15. # 17 
[Sealed] Exhibit 17, # 18 [Sealed] Exhibit 18, #19 [Sealed] Exhibit 19, # 20 
[Sealed] Exhibit 20, # 21 [Sealed] Exhibit 21, #22 [Sealed] Exhibit 22, # 23 
[Sealed] Exhibit 23, # 24 [Sealed] Exhibit 24, # 25 [Sealed] Exhibit 25, # 26 
[Sealed] Exhibit 26, # 27 [Sealed] Exhibit 27, # 28 [Sealed] Exhibit 28, # 29 
[Sealed] Exhibit 29, # 30 [Sealed] Exhibit 30, # 31 [Sealed] Exhibit 32, # 32 
[Sealed] Exhibit 33, # 33 [Sealed] Exhibit 34, #34 [Sealed] Exhibit 38, # 35 
[Sealed] Exhibit 41, # 36 [Sealed] Exhibit 42, # 37 [Sealed] Exhibit 43, # 38 
[Sealed] Exhibit 50, # 39 [Sealed] Exhibit 64, # 40 Certificate of Service) 
(Bridges, Andrew) (Entered: 01/21/2016) 
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01/21/2016 

69 

MOTION for Summary Judgment and Opposition to Plaintiffs' Motion for 
Summary Judgment and Permanent Injunction by PUBLIC.RESOURCE.ORG, 
INC. (Attachments: # \_ Memorandum in Support of Public.Resource.Org's 
Motion for Summary Judgment and Opposition to Plaintiffs' Motion for 
Summary Judgment and Permanent Injunction, # 2 Statement of Material 

Facts, # 3 Statement of Disputed Facts, # 4 Objections to Evidence, # 5 
Declaration of Cal l Malamud, # 6 Declaration of Matthew Becker, # 7 Request 
for Judicial Notice, # 8 Text of Proposed Order Granting Public.Resource.Org's 
Motion for Summary Judgment and Denying Plaintiffs' Motion for Summary 
Judgment and Permanent Injunction)(Bridges, Andrew) (Entered: 01/22/2016) 

01/21/2016 

71 

Memorandum in opposition to re 60 MOTION for Summary Judgment Filed 
MOTION for Permanent Injunction filed by PUBLIC.RESOURCE.ORG, 

INC..(See docket entry no. 69 to view dociunent.) (td) (Entered: 01/22/2016) 

01/22/2016 

70 

LARGE ADDITIONAL ATTACHMENT(S) Index of Consolidated Exhibits In 
Support of Public.Resource.Org's Motion for Summary Judgment and 
Opposition to Plaintiffs' Motion for Summary Judgment and Permanent 
Injunction by PUBLIC.RESOURCE.ORG, INC. 69 MOTION for Summary 
Judgment and Opposition to Plaintiffs' Motion for Summary Judgment and 
Permanent Injunction filed by PUBLIC.RESOURCE.ORG, INC.. 

(Attachments: # l Exhibit 1, # 2 Exhibit 2 [Sealed], # 3 Exhibit 3 [Sealed], # 4 
Exhibit 4 [Sealed], # 5 Exhibit 5 [Sealed], # 6 Exhibit 6 [Sealed], # 7 Exhibit 7, 

# 8 Exhibit 8 [Sealed], # 9 Exhibit 9, # 10 Exhibit 10, # 11 Exhibit 11 [Sealed], 

# 12 Exhibit 12 [Sealed], # J_3 Exhibit 13 [Sealed], # 14 Exhibit 14 [Sealed], # 

15 Exhibit 15 [Sealed], # 16 Exhibit 17 [Sealed], # 17 Exhibit 18 [Sealed], # 18 
Exhibit 19 [Sealed], # 19 Exhibit 20 [Sealed], # 20 Exhibit 21 [Sealed], # 21 
Exhibit 22 [Sealed], # 22 Exhibit 23 [Sealed], # 23 Exhibit 24 [Sealed], # 24 
Exhibit 25 [Sealed], # 25 Exhibit 26 [Sealed], # 26 Exhibit 27 [Sealed], # 27 
Exhibit 28 [Sealed], # 28 Exhibit 29 [Sealed], # 29 Exhibit 30 [Sealed], # 30 
Exhibit 31, # 31 Exhibit 32 [Sealed], # 32 Exhibit 33 [Sealed], # 33 Exhibit 34 
[Sealed], # 34 Exhibit 35, #35 Exhibit 36, # 36 Exhibit 37, # 37 Exhibit 38 
[Sealed], # 38 Exhibit 39, # 39 Exhibit 40, # 40 Exhibit 41 [Sealed], # 41 

Exhibit 42 [Sealed], # 42 Exhibit 43 [Sealed], # 43 Exhibit 44, # 44 Exhibit 45, 

# 45 Exhibit 46, # 46 Exhibit 47, # 47 Exhibit 48, # 48 Exhibit 49, # 49 Exhibit 
50 [Sealed], # 50 Exhibit 51, #51 Exhibit 52, # 52 Exhibit 53, # 53 Exhibit 54, 

# 54 Exhibit 55, # 55 Exhibit 56, # 56 Exhibit 57, # 57 Exhibit 58, # 58 Exhibit 
59, # 59 Exhibit 60, # 60 Exhibit 61, # 61 Exhibit 62, # 62 Exhibit 63, # 63 
Exhibit 64 [Sealed], # 64 Exhibit 65, # 65 Exhibit 66, # 66 Exhibit 67, # 67 
Exhibit 68, # 68 Exhibit 69, # 69 Exhibit 70, # 70 Exhibit 71, # 71 Exhibit 72, 
#72 Exhibit 73, #73 Exhibit 74)(Bridges, Andrew) (Entered: 01/22/2016) 

01/25/2016 

72 

Consent MOTION for Extension of Time to File Response/Reply as to 67 
MOTION to Strike 66 SEALED MOTION FOR LEAVE TO FILE 
DOCUMENT UNDER SEAL filed by PUBLIC.RESOURCE.ORG, INC. 

(This document is SEALED and only available to authorized persons.) filed by 
AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC., 
AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL 
COUNCIL ON MEASUREMENT IN EDUCATION, INC. (Attachments: # 1 
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Text of Proposed Order, # 2 Certificate of Service)(Hudis, Jonathan) (Entered: 
01/25/2016) 

01/28/2016 

73 

Consent MOTION for Leave to Appear Pro Hac Vice : Attorney Name- Nikia 

L. Gray, :Firm- Quarles & Brady LLP, :Address- 1700 K Street, NW, Ste 825. 
Phone No. - 202-372-9600. Fax No. - 202-372-9599 Filing fee $ 100, receipt 
number 0090-4392805. Fee Status: Fee Paid, by AMERICAN 

EDUCATIONAL RESEARCH ASSOCIATION, INC., AMERICAN 
PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC. (Attachments: # J_ Declaration of 
Nikia L. Gray, # 2 Text of Proposed Older for Admission of Nikia L. Gray Pro 
Hac Vice, # 3 Certificate of Service)(Hudis, Jonathan) (Entered: 01/28/2016) 

01/29/2016 


MINUTE ORDER: Granting 73 Motion for Leave to Appear Pro Hac Vice. 
Attorney Nikia L. Gray is hereby admitted pro hac vice to appear in this matter 
on behalf of Plaintiffs American Educational Research Association, Inc., 
American Psychological Association, Inc., and National Coimcil on 
Measurement in Education, Inc. Signed by Judge Tanya S. Chutkan on 

1/26/16. (DJS) (Entered: 01/29/2016) 

01/29/2016 


MINUTE ORDER: Granting 72 Plaintiffs' Consent Motion for Extension of 
Time. Plaintiffs shall respond to Defendant's motion to strike the Declaration 
of Kurt P. Geisinger 67 by February 18, 2016. Defendant's reply due March 3, 
2016. Signed by Judge Tanya S. Chutkan on 1/29/16. (DJS) (Entered: 
01/29/2016) 

02/04/2016 


MINUTE ORDER granting 66 Sealed Motion for Leave to File Document 
Under Seal; granting 68 Sealed Motion for Leave to File Document Under 

Seal. Signed by Judge Tanya S. Chutkan on 2/4/16. (zsm) (Entered: 

02/04/2016) 

02/04/2016 

74 

SEALED DOCUMENT filed by PUBLIC.RESOURCE.ORG, INC., re Older 
on Sealed Motion for Leave to File Document Under Seal,. (This document is 
SEALED and only available to authorized persons.)(ztd) (Entered: 02/04/2016) 

02/04/2016 

75 

SEALED DOCUMENT filed by PUBLIC.RESOURCE.ORG, INC., re Order 
on Sealed Motion for Leave to File Document Under Seal,. (This document is 
SEALED and only available to authorized persons.)(ztd) (Entered: 02/04/2016) 

02/08/2016 


MINUTE ORDER: Granting 63 Motion for Leave to Appeal' Pro Hac Vice. 
Attorney Sebastian E. Kaplan is hereby admitted pro hac vice to appear in this 
matter on behalf of Defendant. Signed by Judge Tanya S. Chutkan on 2/8/16. 
(DJS) (Entered: 02/08/2016) 

02/11/2016 

76 

NOTICE of Appearance by Bruce D. Brown on behalf of The Reporters 
Committee for Freedom of the Press (Brown, Bruce) (Main Document 76 
replaced on 2/11/2016) (td). (Entered: 02/11/2016) 

02/11/2016 

21 

Consent MOTION for Leave to File Amicus Curiae Brief by The Reporters 
Committee for Freedom of the Press (Attachments: # 1 Proposed Amicus 

Ciuiae Brief, # 2 Text of Proposed Order)(Brown, Brace) (Entered: 

02/11/2016) 
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02/11/2016 

78 

Unopposed MOTION for Leave to File Amicus Brief by Law Scholars 
(Attachments: # 1 Exhibit BRIEF OF LAW SCHOLARS AS AMICI CURIAE 
IN SUPPORT OF DEFENDANTCOUNTERCLAIMANT, # 2 Text of 

Proposed Order)(Gellis, Catherine) (Entered: 02/11/2016) 

02/11/2016 

79 

NOTICE of Appearance by Charles Duan on behalf of PUBLIC 

KNOWLEDGE (Duan, Charles) (Entered: 02/11/2016) 

02/11/2016 

80 

LCvR 7.1 CERTIFICATE OF DISCLOSURE of Corporate Affiliations and 
Financial Interests by PUBLIC KNOWLEDGE (Duan, Charles) (Entered: 
02/11/2016) 

02/11/2016 

81 

MOTION for Leave to File Amicus Curiae Brief by PUBLIC KNOWLEDGE 
(Attachments: # I Exhibit, # 2 Text of Proposed Order)(Duan, Charles) 

(Entered: 02/11/2016) 

02/11/2016 

82 

NOTICE of Appearance by Jeffrey T. Pearlman on behalf of SINA BAHRAM 
(Pearlman, Jeffrey) (Entered: 02/11/2016) 

02/11/2016 

83 

Unopposed MOTION for Leave to File Brief of Amicus Curiae in Support of 
Defendant by SINA BAHRAM (Attachments: # 1 Exhibit Proposed Amicus 
Curiae Brief of Sina Bahrain in Support of Defendant, # 2 Text of Proposed 
Order)(Pearlman, Jeffrey) (Entered: 02/11/2016) 

02/12/2016 

84 

MOTION for Leave to File Corrected Declarations by AMERICAN 
EDUCATIONAL RESEARCH ASSOCIATION, INC., AMERICAN 
PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC. (Attachments: # 1 Exhibit 1- 
Declaration of Marianne Ernesto, # 2 Exhibit 2- Declaration of Wayne Camara, 

# 3 Exhibit 3- Declar ation of Felice J. Levine, # 4 Text of Proposed Order, # 5 
Certificate of Service)(Hudis, Jonathan) (Entered: 02/12/2016) 

02/16/2016 


MINUTE ORDER: Granting 77 Motion By Reporters Committee for Freedom 
of the Press for leave to file Amicus brief in support of Defendant's Motion for 
Summary Judgment; Granting 78 Motion By Law Scholars for leave to file 
Amicus brief in support of Defendant's Motion for Summary Judgment; 

Granting 8J_ Motion by Public Knowledge for leave to file Amicus brief in 
support of Defendant’s Motion for Summary Judgment; Granting 83 Motion by 
Sina Bahrain for Leave to file Amicus brief in support of Defendant's Motion 
for Summary Judgment. Signed by Judge Tanya S. Chutkan on 2/16/16. (DJS) 
(Entered: 02/16/2016) 

02/16/2016 


MINUTE ORDER: Granting 84 Plaintiffs' Consent Motion for Leave to File 
corrected Declarations. The Declarations of Marianne Ernesto, Wayne Camara, 
and Felice Levine (ECF Nos. 60-49, 60-76, and 60-78) filed in Support of 
Plaintiffs' Motion for Summary Judgment and a Permanent Injunction 60 are 
hereby accepted for filing without the necessity of further filing or serving any 
copy. Signed by Judge Tanya S. Chutkan on 2/16/16. (DJS) (Entered: 
02/16/2016) 

02/16/2016 

85 

AMICUS BRIEF by REPORTERS COMMITTEE FOR FREEDOM OF THE 
PRESS, (td) Modified date filed on 2/17/2016 (td). (Entered: 02/17/2016) 
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02/16/2016 

86 

AMICUS BRIEF by Law Scholars, (td) (Entered: 02/17/2016) 

02/16/2016 

87 

AMICUS BRIEF by PUBLIC KNOWLEDGE, (td) (Entered: 02/17/2016) 

02/16/2016 

88 

AMICUS BRIEF by SINA BAHRAM. (td) (Entered: 02/17/2016) 

02/18/2016 

89 

REPLY to opposition to motion re 69 MOTION for Summary Judgment and 
Opposition to Plaintiffs' Motion for Summary Judgment and Permanent 
Injunction , 60 MOTION for Summary Judgment Filed MOTION for 

Permanent Injunction (Plaintiffs' Reply in Further Support of Plaintiffs' Motion 
for Summary Judgment and Permanent Injunction and Opposition to 

Defendant’s Motion for Summary Judgment) filed by AMERICAN 
EDUCATIONAL RESEARCH ASSOCIATION, INC., AMERICAN 
PSYCHOLOGICAL ASSOCIATION, INC.. NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC.. (Attachments: # 1 Plaintiffs' 
Statement of Material Facts in Opposition to Defendant's Motion for Summary 
Judgment, # 2 Plaintiffs' Reply to Defendants' Statement of Disputed Facts, # 3 
Plaintiffs' Response to Defendant's Objections to Plaintiffs' Evidence, # 4 
Declaration of Nikia L. Gray, # 5 Exhibit VW, # 6 Exhibit WWW, # 7 

Exhibit XXX, # 8 Exhibit YYY, # 9 Exhibit ZZZ, # K) Exhibit AAAA, # JT 
Exhibit BBBB, # 12 Exhibit CCCC, # 13 Exhibit DDDD, # 14 Exhibit EEEE, 

# 15 Exhibit FFFF, # 16 Exhibit GGGG, # 17 Exhibit HHHH, # 18 Exhibit IIH, 

# 19 Exhibit JJJJ, # 20 Exhibit KKKK, # 21 Exhibit LLLL, # 22 Exhibit 
MMMM, # 23 Exhibit NNNN, # 24 Exhibit OOOO, # 25 Exhibit PPPP, # 26 
Exhibit QQQQ, # 27 Exhibit RRRR, # 28 Exhibit SSSS, # 29 Exhibit TTTT, # 
30 Exhibit UUUU, # 31 Exhibit WW, # 32 Exhibit WWWW, # 33 Exhibit 
XXXX, # 34 Exhibit YYYY, # 35 Exhibit ZZZZ, # 36 Exhibit AAAAA, # 37 
Exhibit BBBBB, # 38 Exhibit CCCCC, # 39 Exhibit DDDDD, # 40 Exhibit 
EEEEE, # 41 Exhibit FFFFF, # 42 Exhibit GGGGG, # 43 Exhibit HHHHH, # 

44 Exhibit imi, # 45 Exhibit JJJJJ, # 46 Exhibit KKKKK, # 47 Exhibit 

LLLLL, # 48 Exhibit MMMMM, # 49 Exhibit NNNNN, # 50 Exhibit 

OOOOO, # 51 Exhibit PPPPP, # 52 Exhibit QQQQQ, # 53 Exhibit RRRRR, # 
54 Exhibit SSSSS, # 55 Exhibit TTTTT, # 56 Exhibit UUUUU, # 57 Exhibit 
VWW, # 58 Exhibit WWWWW, # 59 Exhibit XXXXX, # 60 Exhibit 
YYYYY, # 61 Exhibit ZZZZZ, # 62 Exhibit AAAAAA, # 63 Declaration of 
Wayne Camara, # 64 Plaintiffs' Objections to Defendant's Evidence, # 65 
Certificate of Service)(Hudis, Jonathan) (Entered: 02/18/2016) 

02/18/2016 

90 

SEALED MOTION FOR LEAVE TO FILE DOCUMENT UNDER SEAL 
filed by AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC., 
AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL 
COUNCIL ON MEASUREMENT IN EDUCATION, INC. (This document is 
SEALED and only available to authorized persons.) (Attachments: # J_ 
Memorandum in Support, # 2 Plaintiffs' Statement of Genuine, Disputed Issues 
of Material Facts, # 3 Plaintiffs' Reply to Defendant's Statement of Disputed 
Facts, # 4 Plaintiffs' Response to Defendants' Objections to Plaintiffs' 

Evidence, # 5 Exhibit VWW - Expert Report of S.E. Phillips, Ph.D., J.D., # 

6 Exhibit XXXXX - Deposition Transcript of Wayne J. Camara, # 7 Exhibit 
YYYYY - Deposition Transcript of Dianne L. Schneider, # 8 Exhibit 

AAAAAA - Deposition Transcript of Marianne Ernesto, # 9 Plaintiffs' 
Opposition to Defendant's Motion to Strike the Declar ation of Kurt F. 
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Geisinger (sealed version), # 10 Exhibit 1- Deposition Transcript of Kurt F. 
Geisinger, # JT Text of Proposed Order, # 12 Certificate of Service)(Hudis, 
Jonathan) (Entered: 02/18/2016) 

02/18/2016 

91 

RESPONSE re 67 MOTION to Strike 66 SEALED MOTION FOR LEAVE 

TO FILE DOCUMENT UNDER SEAL filed by PUBLIC.RESOURCE.ORG, 
INC. (This document is SEALED and only available to authorized persons.) 
(Plaintiffs' Opposition to Defendant's Motion to Strike the Declaration of Kurt 

F. Geisinger) filed by AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, 
INC., NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION, 

INC.. (Attachments: # 1 Declaration of Jonathan Hudis, # 2 Exhibit 1, # 3 
Certificate of Service)(Hudis, Jonathan) (Entered: 02/18/2016) 

02/18/2016 


MINUTE ORDER granting 90 Sealed Motion for Leave to File Document 
Under Seal. Signed by Judge Tanya S. Chutkan on 2/18/16. (zsm) (Entered: 
02/18/2016) 

02/18/2016 

92 

SEALED DOCUMENT filed by AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, 
INC., NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION, 

INC., re Order on Sealed Motion for Leave to File Document Under Seal. (This 
document is SEALED and only available to authorized persons.)(ztd) (Entered: 
02/19/2016) 

02/18/2016 

93 

SEALED REPLY TO OPPOSITION filed by AMERICAN EDUCATIONAL 
RESEARCH ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL 
ASSOCIATION, INC., NATIONAL COUNCIL ON MEASUREMENT IN 
EDUCATION, INC. re 60 MOTION for Summary Judgment Filed MOTION 
for Permanent Injunction (ztd) (Entered: 02/19/2016) 

02/18/2016 

94 

SEALED DOCUMENT filed by AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, 
INC., NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION, 

INC.. (This document is SEALED and only available to authorized persons.) 
(Attachments: # l Exhibit VVVW, # 2 Exhibit XXXXX, # 3 Exhibit 

YYYYY, # 4 Exhibit AAAAAA)(ztd) (Entered: 02/19/2016) 

02/18/2016 

95 

SEALED OPPOSITION filed by AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, 
INC., NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION, INC. 
re 67 MOTION to Strike 66 SEALED MOTION FOR LEAVE TO FILE 
DOCUMENT UNDER SEAL filed by PUBLIC.RESOURCE.ORG, INC. 

(This document is SEALED and only available to authorized persons.) 
(Attachments: # J. Exhibit)(ztd) (Entered: 02/19/2016) 

03/03/2016 

96 

SEALED MOTION FOR LEAVE TO FILE DOCUMENT UNDER SEAL 
filed by PUBLIC.RESOURCE.ORG, INC. (This document is SEALED and 
only available to authorized persons.) (Attachments: # \ Exhibit [Filed Under 
Seal] Reply In Support of Defendant Public Resource's Motion to Strike the 
Declaration of Kurt F. Geisinger [90-12], # 2 Text of Proposed Order Granting 
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Motion to File Under Seal, # 3 Certificate of Service Re Motion to File Under 
Seal)(Bridges, Andrew) (Entered: 03/03/2016) 

03/03/2016 

97 

REPLY to opposition to motion re 67 MOTION to Strike 66 SEALED 

MOTION FOR LEAVE TO FILE DOCUMENT UNDER SEAL filed by 
PUBLIC.RESOURCE.ORG, INC. (This document is SEALED and only 
available to authorized persons.) [REDACTED] REPLY IN SUPPORT OF 
PUBLIC RESOURCE'S MOTION TO STRIKE THE DECLARATION KURT F. 
GEISINGER [90-12] filed by PUBLIC.RESOURCE.ORG, INC.. (Bridges, 
Andrew) (Entered: 03/03/2016) 

03/03/2016 

98 

SEALED MOTION FOR LEAVE TO FILE DOCUMENT UNDER SEAL 
filed by PUBLIC.RESOURCE.ORG, INC. (This document is SEALED and 
only available to authorized persons.) (Attachments: # 1 [Filed Under Seal] 
Reply In Support of Defendant Public Resource's Motion for Summary 
Judgment, #2 [Filed Under Seal] Declaration of Matthew Becker In Support of 
Defendant's Reply to Its Motion for Summary Judgment, # 3 [Filed Under 

Seal] Objections to Plaintiffs' Supplemental Evidence, # 4 Exhibit 75 [Sealed], 

# 5 Exhibit 76 [Sealed], # 6 Exhibit 77 [Sealed], # 7 Exhibit 78 [Sealed], # 8 
Exhibit 79 [Sealed], # 9 Exhibit 80 [Sealed], # JO Text of Proposed Order 
Granting Motion to File Under Seal, # JT Certificate of Service Re Motion to 
File Under Seal)(Bridges, Andrew) (Entered: 03/03/2016) 

03/03/2016 

99 

REPLY to opposition to motion re 69 MOTION for Summary Judgment and 
Opposition to Plaintiffs' Motion for Summary Judgment and Permanent 
Injunction filed by PUBLIC.RESOURCE.ORG, INC.. (Attachments: #J 
[Public Redacted] Reply Declaration of Matthew Becker In Further Support of 
Defendant's Motion for Summary Judgment, # 2 [Public Redacted] Defendant's 
Objections to Plaintiffs' Supplemental Evidence, # 3 Supplemental Request for 
Judicial Notice, # 4 [Public Redacted] Objections to Plaintiffs' Supplemental 
Evidence, # 5 Exhibit 75 [Sealed Placeholder], #6 Exhibit 76 [Sealed 
Placeholder], # 7 Exhibit 77 [Sealed Placeholder], #8 Exhibit 78 [Sealed 
Placeholder], # 9 Exhibit 79 [Sealed Placeholder!. #10 Exhibit 80 [Sealed 
Placeholder], # JT Exhibit 81. # 12 Exhibit 82, # J_3 Exhibit 83)(Bridges, 
Andrew) (Entered: 03/03/2016) 

03/04/2016 


Minute ORDER granting 98 Sealed Motion for Leave to File Document Under 
Seal. Signed by Judge Tanya S. Chutkan on 3/4/16. (zsm) (Entered: 

03/07/2016) 

03/04/2016 

100 

SEALED REPLY TO OPPOSITION filed by PUBLIC.RESOURCE.ORG, 

INC. re 69 MOTION for Summary Judgment and Opposition to Plaintiffs' 
Motion for Summary Judgment and Permanent Injunction, 98 SEALED 
MOTION FOR LEAVE TO FILE DOCUMENT UNDER SEAL filed by 
PUBLIC.RESOURCE.ORG, INC. (This document is SEALED and only 
available to authorized persons.) (ztd) (Entered: 03/07/2016) 

03/08/2016 

101 

NOTICE of Intent to File Oppositions to Defendant's Motions, filed by 
AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC., 
AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL 
COUNCIL ON MEASUREMENT IN EDUCATION, INC. re 99 Reply to 
opposition to Motion,,, (Hudis, Jonathan) (Entered: 03/08/2016) 
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03/15/2016 

102 

SEALED MOTION FOR LEAVE TO FILE DOCUMENT UNDER SEAL 
filed by PUBLIC.RESOURCE.ORG, INC. (This document is SEALED and 
only available to authorized persons.) (Attachments: # 1_ Text of Proposed 

Order Granting Defendant's Unopposed Motion for Leave to File Under Seal, # 

2 Exhibit 5 [Sealed Version] of Corrected Memorandum of Points and 
Authorities In Support of Defendant's Motion for Summary Judgment and 
Opposition to Plaintiffs' Motion for Summary Judgment, # 3 Exhibit 6 [Sealed 
Version] of Collected Statement of Material Facts In Support of Defendant 
Public Resource's Motion for Summary Judgment and Opposition to Plaintiffs' 
Motion for Summary Judgment and Permanent Injunction, # 4 Exhibit 7 
[Sealed Version] of Corrected Statement of Disputed Facts in Opposition to 
Plaintiffs' Motion for Summary Judgment, # 5 Certificate of Service Re Motion 
to Seal)(Becker, Matthew) (Entered: 03/15/2016) 

03/15/2016 

103 

Unopposed MOTION for Leave to File Corrected Documents by 
PUBLIC.RESOURCE.ORG, INC. (Attachments: # 1 Text of Proposed Order 
Granting Defendant's Unopposed Motion for Leave to File Collected 
Documents, # 2 Exhibit 1 [Redacted Version] of Memorandum of Points and 
Authorities In Support of Defendant Public Resource's Motion for Summary 
Judgment and Opposition to Plaintiffs' Motion for Summary Judgment and 
Permanent Injunction, # 3 Exhibit 2 [Redacted Version] of Statement of 

Material Facts In Support of Defendant Public Resource's Motion for Summary 
Judgment and Opposition to Plaintiffs' Motion for Summary Judgment and 
Permanent Injunction, # 4 Exhibit 3 [Redacted Version] of Statement of 
Disputed Facts In Opposition to Plaintiffs' Motion for Summary Judgment and 
Permanent Injunction, # 5 Exhibit 4 Table of Corrections)(Becker, Matthew) 
(Entered: 03/15/2016) 

03/17/2016 


MINUTE ORDER granting 103 Motion for Leave to File. Signed by Judge 
Tanya S. Chutkan on 3/17/16. (zsm) (Entered: 03/17/2016) 

03/17/2016 

104 

SEALED DOCUMENT filed by PUBLIC.RESOURCE.ORG, INC., re Order 
on Sealed Motion for Leave to File Document Under Seal. (This document is 
SEALED and only available to authorized persons.)(ztd) (Entered: 03/17/2016) 

03/17/2016 

105 

REDACTED DOCUMENT- to 69 MOTION for Summary Judgment and 
Opposition to Plaintiffs' Motion for Summary Judgment and Permanent 
Injunction, Order on Motion for Leave to Fiie by PUBLIC.RESOURCE.ORG, 
INC. (td) (Entered: 03/17/2016) 

03/21/2016 

106 

SEALED MOTION FOR LEAVE TO FILE DOCUMENT UNDER SEAL 
filed by AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC., 
AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL 
COUNCIL ON MEASUREMENT IN EDUCATION, INC. (This document is 
SEALED and only available to authorized persons.) (Attachments: # J. 
Memorandum in Support, # 2 Exhibit 1 to Declaration of Jonathan Hudis, # 3 
Plaintiffs' Objections to Defendant's Evidence in Support of its Reply 
Memorandum, # 4 Text of Proposed Order, # 5 Certificate of Service)(Hudis, 
Jonathan) (Entered: 03/21/2016) 

03/21/2016 

107 
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Memorandum in opposition to re 98 SEALED MOTION FOR LEAVE TO 

FILE DOCUMENT UNDER SEAL filed by PUBLIC.RESOURCE.ORG, 

INC. (This document is SEALED and only available to authorized persons.) 
filed by AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC., 
AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL 
COUNCIL ON MEASUREMENT IN EDUCATION, INC.. (Attachments: # 1 
Declaration of Jonathan Hudis, # 2 Certificate of Service)(Hudis, Jonathan) 
(Entered: 03/21/2016) 

03/21/2016 

108 

RESPONSE re 98 SEALED MOTION FOR LEAVE TO FILE DOCUMENT 
UNDER SEAL filed by PUBLIC.RESOURCE.ORG, INC. (This document is 
SEALED and only available to authorized persons.) Plaintiffs' Responses to 
Defendant's Objections to Plaintiffs' Supplemental Evidence filed by 
AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC., 
AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL 
COUNCIL ON MEASUREMENT IN EDUCATION, INC.. (Hudis, Jonathan) 
(Entered: 03/21/2016) 

03/21/2016 

109 

RESPONSE re 98 SEALED MOTION FOR LEAVE TO FILE DOCUMENT 
UNDER SEAL filed by PUBLIC.RESOURCE.ORG, INC. (This document is 
SEALED and only available to authorized persons.) Plaintiffs' Objections to 
Defendant-Counterclaimant Public.Resource.Org, Inc. 's Evidence In Support 
of Defendant-Counterclaimant's Reply Memorandum in Support of Its Motion 
for Summary Judgment filed by AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, 
INC., NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION, 

INC.. (Hudis, Jonathan) (Entered: 03/21/2016) 

03/21/2016 

110 

SEALED DOCUMENT filed by AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, 
INC., re 99 Reply to opposition to Motion,,, filed by 
PUBLIC.RESOURCE.ORG, INC.. (This document is SEALED and only 
available to authorized persons.) (Attachments: # _1 Exhibit)(ztd) (Entered: 
03/22/2016) 

03/22/2016 


MINUTE ORDER granting 106 Sealed Motion for Leave to File Document 
Under Seal. Signed by Judge Tanya S. Chutkan on 3/22/16. (zsm) (Entered: 
03/22/2016) 

03/31/2016 

111 

SEALED MOTION FOR LEAVE TO FILE DOCUMENT UNDER SEAL 
filed by PUBLIC.RESOURCE.ORG, INC. (This document is SEALED and 
only available to authorized persons.) (Attachments: # \ Text of Proposed 

Order Granting Motion to Seal Defendant Public Resource's Reply to Its 
Objections and Responses to Plaintiffs' Objections, # 2 Certificate of Service 

Re Motion to Seal Defendant Public Resource's Reply to Its Objections and 
Responses to Plaintiffs' Objections, #3 [Filed Under Seal] Defendant 

Pub lie.Resource.Org, Inc.'s Reply to Its Objections and Motions to Strike 
Plaintiffs' Supplemental Evidence, # 4 [Filed Under Seal] Responses to 

Plaintiffs' Objections to Defendant's Evidence in Support of Its Reply 
Memorandum In Support of Its Motion for Summary Judgment)(Becker, 
Matthew) (Entered: 03/31/2016) 
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03/31/2016 

112 

REPLY to opposition to motion re .106 SEALED MOTION FOR LEAVE TO 
FILE DOCUMENT UNDER SEAL filed by AMERICAN EDUCATIONAL 
RESEARCH ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL 
ASSOCIATION, INC., NATIONAL COUNCIL ON MEASUREMENT IN 
EDUCATION, INC. (This document is SEALED and only available to aut 
[Redacted Version] Defendant Public.Resource. Org, Inc. 's Reply to Its 
Objections and Motions to Strike Plaintiffs' Supplemental Evidence filed by 
PUBLIC.RESOURCE.ORG, INC.. (Attachments: # 1 Declaration of Matthew 
Becker In Support of Defendant Public Resource's Reply to Its Objections and 
Motions to Strike Plaintiffs' Supplemental Evidence)(Becker, Matthew) 

(Entered: 03/31/2016) 

03/31/2016 

113 

RESPONSE re 106 SEALED MOTION FOR LEAVE TO FILE DOCUMENT 
UNDER SEAL filed by AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, 
INC., NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION, INC. 
(This document is SEALED and only available to aut [Redacted Version] 
Defendant Public.Resource.Org, Inc. 's Responses to Plaintiffs' Objections to 
Defendant’s Evidence In Support of Its Reply Memorandum In Support of Its 
Motion for Summary Judgment filed by PUBLIC.RESOURCE.ORG, INC.. 
(Becker, Matthew) (Entered: 03/31/2016) 

05/26/2016 

114 

NOTICE of Change of Address by Jonathan Hudis (Hudis, Jonathan) (Entered: 
05/26/2016) 

06/03/2016 


MINUTE ORDER. Motion Heating on all pending motions set for 9/12/2016 
at 9:30 AM in Courtroom 2 before Judge Tanya S. Chutkan. Signed by Judge 
Tanya S. Chutkan on 6/3/16. (Ictsc2) (Entered: 06/03/2016) 

06/03/2016 


Set/Reset Hearings: Motion Hearing set for 9/12/2016 at 9:30 AM in 

Courtroom 2 before Judge Tanya S. Chutkan. (zsm) (Entered: 06/03/2016) 

09/09/2016 


MINUTE ORDER: The motions hearing previously scheduled for 9:30 a.m. on 
9/12/2016 has been rescheduled to begin at 9:00 a.m. in Courtroom 2. Signed 
by Judge Tanya S. Chutkan on 9/9/2016. (Ictsc2) (Entered: 09/09/2016) 

09/09/2016 


Set/Reset Hearings: Motion Hearing set for 9/12/2016 at 9:00 AM in 

Courtroom 2 before Judge Tanya S. Chutkan. (zsm) (Entered: 09/09/2016) 

09/12/2016 


Minute Entry for proceedings held before Judge Tanya S. Chutkan: Motion 
Healing held on 9/12/2016 re 69 MOTION for Summary Judgment and 
Opposition to Plaintiffs' Motion for Summary Judgment and Permanent 
Injunction filed by PUBLIC.RESOURCE.ORG, INC., 60 MOTION for 
Summary Judgment Filed MOTION for Permanent Injunction filed by 
AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC., 
AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL 
COUNCIL ON MEASUREMENT IN EDUCATION, INC. Motions taken 
under advisement. (Court Reporter Bryan Wayne.) (zsm) (Entered: 

09/12/2016) 

09/21/2016 

115 

ORDER denying 67 Defendant's Motion to Strike Expert Declaration. Signed 
by Judge Tanya S. Chutkan on 9/21/2016. (Ictsc2) (Entered: 09/21/2016) 
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10/13/2016 

116 

TRANSCRIPT OF 9/12/16 MOTIONS HEARING, before Judge Tanya S. 
Ckutkan, held on September 12, 2016. Page Numbers: 1-142. Date of Issuance: 
10/13/16. Court Reporter: Bryan A. Wayne. Transcripts may be ordered by 
submitting the Transcript Older Form 

For the first 90 days after this filing date, the transcript may be viewed at the 
courthouse at a public terminal or purchased from the court reporter referenced 
above. After 90 days, t he transcript may be accessed via PACER. Other 
transcript formats, (multi-page, condensed, CD or ASCII) may be purchased 
from the court reporter. 

NOTICE RE REDACTION OF TRANSCRIPTS: The parties have twenty- 
one days to file with the court and the court reporter any request to redact 
personal identifiers from this transcript. If no such requests are filed, the 
tr anscript will be made available to the public via PACER without redaction 
after 90 days. The policy, which includes the five personal identifiers 
specifically covered, is located on our website at www.dcd.uscom1s.gov. 

Redaction Request due 11/3/2016. Redacted Transcript Deadline set for 

11/13/2016. Release of Transcript Restriction set for 1/11/2017.(Wayne, 

Bryan) (Entered: 10/13/2016) 

02/02/2017 

117 

MEMORANDUM AND OPINION re 60 Plaintiffs' motion for summary 
judgment and 69 Defendant's cross-motion for summary judgment. Signed by 
Judge Tanya S. Chutkan on 2/2/2017. (Ictsc2) (Entered: 02/02/2017) 

02/02/2017 

118 

ORDER granting in part and denying in part 60 Plaintiffs' motion for summary 
judgment; denying 69 Defendant's cross-motion for summary judgment; 
denying as moot 96 Sealed Motion for Leave to File Document Under Seal; 
denying as moot 1JT Sealed Motion for Leave to File Document Under Seal. 
Signed by Judge Tanya S. Chutkan on 2/2/2017. (Ictsc2) (Entered: 02/02/2017) 

02/10/2017 

119 

MOTION to Clari fyOrder [DktNo 118] and, in Alternative, for Continuance 
by AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC., 
AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL 
COUNCIL ON MEASUREMENT IN EDUCATION, INC. (Attachments: # 1 
Exhibit A, # 2 Text of Proposed Order, # 3 Certificate of Service)(Hudis, 
Jonathan) (Entered: 02/10/2017) 

02/13/2017 


MINUTE ORDER: 119 Plaintiffs' Motion is GRANTED to the extent that 
Plaintiffs are not required to abide by the deadline for requesting attorney fees 
under Federal Rule 54(d). The parties are hereby ORDERED to file a joint 
status report by 3/3/2017 (1) updating the court as to Defendant's compliance 
with 118 the court's order to remove the standards from its website and (2) 
providing a jointly proposed schedule for moving forward with this litigation. 
Signed by Judge Tanya S. Chutkan on 2/13/2017. (Ictsc2) (Entered: 

02/13/2017) 

02/14/2017 


Set/Reset Deadlines: Joint Status Report due by 3/3/2017. (jth) (Entered: 
02/14/2017) 

02/17/2017 

120 
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NOTICE OF APPEAL TO DC CIRCUIT COURT as to 118 Order on Motion 
for Summary Judgment,, Order on Motion for Permanent Injunction,,,, Order 
on Sealed Motion for Leave to File Document Under Seal,,, 117 Memorandum 
& Opinion by PUBLIC.RESOURCE.ORG, INC.. Filing fee $ 505, receipt 
number 0090-4848021. Fee Status: Fee Paid. Parties have been notified. 
(Bridges, Andrew) (Entered: 02/17/2017) 

02/21/2017 

121 

Transmission of the Notice of Appeal, Order Appealed, and Docket Sheet to 

US Comt of Appeals. The Court of Appeals fee was paid this date re 120 

Notice of Appeal to DC Circuit Court,, (jf) (Entered: 02/21/2017) 

03/02/2017 


USCA Case Number 17-7039 for 120 Notice of Appeal to DC Circuit Court, 
filed by PUBLIC.RESOURCE.ORG, INC. (zrdj) (Entered: 03/02/2017) 

03/03/2017 

122 

Joint STATUS REPORT in compliance with Court's Minute order of Feb. 13, 
2017, filed by AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, 
INC., AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL 
COUNCIL ON MEASUREMENT IN EDUCATION, INC.. (Attachments: # \ 
Certificate of Service)(Hudis, Jonathan) (Entered: 03/03/2017) 

03/03/2017 

123 

Consent MOTION to Stay re 122 Status Report, pending appeal to DC Circuit, 
filed by AMERICAN EDUCATIONAL RESEARCH ASSOCIATION, INC., 
AMERICAN PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL 
COUNCIL ON MEASUREMENT IN EDUCATION, INC. (Attachments: # \ 
Text of Proposed Order, # 2 Certificate of Service)(Hudis, Jonathan) (Entered: 
03/03/2017) 

03/03/2017 


MINUTE ORDER granting 123 Plaintiffs' stipulated motion to stay 
proceedings in this case pending resolution of Defendant's appeal. Signed by 
Judge Tanya S. Chutkan on 3/3/2017. (Ictsc2) (Entered: 03/03/2017) 

03/03/2017 

124 

NOTICE RE PRELIMINARY AND NON-BINDING STATEMENT OF ISSUES 
BY APPELLANT/DEFENDANT-COUNTERCLAIMANT 
PUBLICRESOURCE.ORG, INC. by PUBLIC.RESOURCE.ORG, INC. 
(Bridges, Andrew) (Entered: 03/03/2017) 

03/03/2017 

125 

NOTICE OF TRANSCRIPT ORDER BY DEFENDANT- 
COUNTERCLAIMANT PUBLIC.RESOURCE.ORG, INC. by 
PUBLIC.RESOURCE.ORG, INC. (Bridges, Andrew) (Entered: 03/03/2017) 

03/23/2017 

126 

NOTICE of Appear ance by Clifton Scott Elgarten on behalf of AMERICAN 
EDUCATIONAL RESEARCH ASSOCIATION, INC., AMERICAN 
PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC. (Elgarten, Clifton) (Entered: 
03/23/2017) 

03/24/2017 

127 

NOTICE OF WITHDRAWAL OF APPEARANCE as to AMERICAN 
EDUCATIONAL RESEARCH ASSOCIATION, INC., AMERICAN 
PSYCHOLOGICAL ASSOCIATION, INC., NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC.. Attorney Jonathan P. Labukas; 
Nikia L. Gray and Jonathan Hudis terminated. (Hudis, Jonathan) (Entered: 
03/24/2017) 
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07/06/2017 


MINUTE ORDER: In light of the parties' pending appeal before the Circuit 
Court, the Clerk of the Court is hereby directed to Administratively Close this 
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UNITED STATES DISTRICT COURT 
FOR THE DISTRICT OF COLUMBIA 


AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC. 

1430 K Street, NW 
Suite 1200 

Washington, DC 20005, 

AMERICAN PSYCHOLOGICAL 
ASSOCIATION, INC. 

750 First Street NE 
Washington, DC 20002-4242, and 

NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC. 
2424 American Lane 
Madison, WI 53704, 

Plaintiffs, 


PUBLIC.RESOURCE.ORG, INC. 
1005 Gravenstein Highway North 
Sebastopol, CA 95472 

Defendant. 


) 

) 

) 

) 

) 

) 

) Civil Action No._ 

) 

) COMPLAINT FOR COPYRIGHT 
) INFRINGEMENT AND 
) CONTRIBUTORY COPYRIGHT 
) INFRINGEMENT 

) 

) 

) 

) 

) 

) 

) 

) 

) 

) 

) 

) 

) 


COMPLAINT 


Plaintiffs, as and for their Complaint against Defendant, allege as follows: 

1. This is an action by three non-profit organizations: the American Educational 
Research Association, Inc., the American Psychological Association, Inc., and the National 
Council on Measurement in Education, Inc. (collectively, “Plaintiffs”), creators of the work 
entitled “Standards for Educational and Psychological Testing” (the “Standards”). This action 
seeks injunctive relief against Public.Resource.Org, Inc. for infringement and contributory 
infringement of Plaintiffs’ copyright in the Standards. 
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THE PARTIES 

2. Plaintiff, American Educational Research Association, Inc. (“AERA”), is a 
District of Columbia not-for-profit corporation whose main offices are located at 1430 K Street, 
NW, Suite 1200, Washington, DC 20005. 

3. AERA is the major national scientific society for research on education and 
learning. AERA’s mission is to advance knowledge about education, to encourage scholarly 
inquiry related to education, and to promote the use of research to improve education and serve 
the public good. 

4. Plaintiff, American Psychological Association, Inc. (“APA”), is a District of 
Columbia not-for-profit corporation whose main offices are located at 750 First Street, NE, 
Washington, DC 20002. 

5. APA is the largest scientific and professional organization representing 
psychology in the United States. APA is the world's largest association of psychologists and 
counts a vast number of researchers, educators, clinicians, consultants and students among its 
members. APA’s mission is to advance the creation, communication and application of 
psychological knowledge to benefit society and improve people’s lives. 

6. Plaintiff, National Council on Measurement in Education, Inc. (“NCME”), is a 
District of Columbia not-for-profit corporation whose main offices are located at 2424 American 
Lane, Madison, WI 53704. 

7. NCME is a professional organization for individuals involved in assessment, 
evaluation, testing, and other aspects of educational measurement. NCME’s members are 
involved in the construction and use of standardized tests; new forms of assessment, including 
performance-based assessment; program design; and program evaluation. NCME’s members 
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include university faculty; test developers; state and federal testing and research directors; 
professional evaluators; testing specialists in business, industry, education, community programs, 
and other professions; licensure, certification, and credentialing professionals; graduate students 
from educational, psychological, and other measurement programs; and others involved in 
testing issues and practices. 

8. Upon information and belief, Defendant, Public.Resource.Org, Inc. (“Public 
Resource”), is a California not-for-profit corporation having offices located at 1005 Gravenstein 
Highway, North Sebastopol, CA 95472. 

9. According to its Articles of Incorporation, which can be found at 

https://public.resource.org/public.resource.articles.html, the purpose of Public Resource is to 

create, architect, design, implement, operate and maintain public works projects 
on the Internet for Educational, Charitable, and Scientific Purposes to the 
benefit of the general public and the public interest; to increase and diffuse 
knowledge about the Internet in its broadest sense; to promote and facilitate the 
expansion, development, and growth of the public infrastructure of the Internet by 
any means consistent with the public interest through other activities, including, 
but not limited to, publications, meetings, conferences, training, educational 
seminars, and the issuance of grants and other financial support to educational 
institutions, foundations and other organizations exclusively for Educational, 
Charitable, and Scientific Purposes. 

10. Upon information and belief, Public Resource publishes on the Internet, and in 
particular on the website https://law.resource.org, inter alia, state and municipal codes, public 
safety codes, and technical standards. 

11. Upon information and belief, at least some of the codes and standards that Public 
Resource publishes on the Internet, and in particular on the website https://law.resource.org, are 
subject to U.S. copyright protection. 

12. Upon information and belief, Public Resource publishes these codes and 
standards on the Internet, and in particular on the website https://law.resource.org, without 
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obtaining the permission of the copyright owners of these works. 

13. AERA and APA, two of the three joint copyright owners of the Standards, are 
located in Washington, DC. 

14. Public Resource has designed the website 

https://law.resource.org/pub/us/cff/manifest.us.html to attract visitors from the District of 
Columbia to copy, distribute to others, or incorporate into other works, Plaintiffs Standards. 

15. Public Resource has designed the website https://law.resource.org/pub/us/code/dc/ 
to attract visitors from the District of Columbia by providing online copies of the District of 
Columbia Code. 

16. On the website https://public.resource.org/about/index.html. Public Resource 
solicits donations from the public to support Public Resource’s activities, via credit card or 
through Public Resource’s PayPal account. Upon information and belief, Public Resource has 
received donations to support its activities from Google, Yahoo!, the Electronic Frontier 
Foundation, Creative Commons Corporation, Justia Inc., Fenwick & West FFP, Durie Tangri, 
Davis Wright Tremaine FFP, iRights Faw, Alston & Bird, the Elbaz Family Foundation, the 
Cutts Foundation, the O'Reilly Foundation, the Beal Fund, the Sunlight Foundation, the Omidyar 
Network, and others. 

17. Public Resource, through the website https://public.resource.org, solicits visitors 
from the District of Columbia to submit financial donations to Public Resource. 

18. Public Resource, through the website https://public.resource.org/about.index.html, 
allows visitors from the District of Columbia to e-mail Public Resource directly at the e-mail 
address carl@media.org . 

19. Upon information and belief, Public Resource has collaborated with organizations 
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in Washington, DC to copy videos and upload them to YouTube. National Technical 
Information Service Library of Commerce, https://public.resource.org/ntis.gov/ (last visited Apr. 
25, 2014); see also Kate Murphy, Catching Up With Carl Malamud, N.Y. Times, Jan. 26, 2013, 
www.nytimes.com/2013/01/27/opinion/sundav/catching-up-with-carl-malamud.html . 

20. Upon information and belief, Public Resource, through its President and founder 
Carl Malamud, has testified in Washington, DC before the Subco mm ittee on Courts, Intellectual 
Property, and the Internet of the House Judiciary Committee regarding the scope of protection to 
be afforded copyrighted works. An Edict of Government Amendment: Hearing on the Scope of 
Copyright Protection Before the Subcomm. On Courts, Intellectual Property, and the Internet of 
the H. Comm, on the Judiciary, 113th Cong. (2014) (statement of Carl Malamud, President, 
Public.Resource.Org.). 

21. Upon information and belief, Public Resource, through its President and founder 
Carl Malamud, has participated in conferences and given speeches in Washington, DC since at 
least 2009. Peter Glaskowsky, Carl Malamud’s digital manifesto, CNET News, Sept. 16, 2009, 
http://news.cnet.com/8301-13512 3-10354324-23.html; Carl Malamud, gov 2.0 Expo, 
www.gov2expo.com/gov2expo2010/public/schedule/speaker/1824 (last visited Apr. 25, 2014); 
An Evening with Carl Malamud, Next Tuesday at ALA’s Washington DC HQ, Resource Shelf, 

http://web.resourceshelf.com/go/resourceblog/58874 (last visited Apr. 25, 2014). 

22. Defendant has designed its website(s) to encourage visitors from the District of 
Columbia to copy, to distribute to others in this District and/or to create derivative works based 
on Plaintiffs’ Standards. 

23. Defendant also sells various items on the Internet, including Public.Resource.Org 
mouse pads, stamps, and stickers, among other items. See 
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http://www.zazzle.com/carlmalamud/gifts . This website is available to residents of the District 
of Columbia, who may purchase Defendants’ merchandise when visiting the site. 

JURISDICTION AND VENUE 

24. This Court has jurisdiction over this action pursuant to 28 U.S.C. §§ 1331 and 
1338(a). This is a civil action arising under an Act of Congress relating to copyrights. 

25. Venue is proper in this Court pursuant to 28 U.S.C. §§ 1391 and 1400(a). The 
Claims plaintiffs allege in this Complaint arose, in substantial part, in the District of Columbia. 
Defendant may be found in Washington, DC, and this Court has personal jurisdiction over 
Defendant. Defendant has directed its infringing activities to Washington, DC and, on 
information and belief, materially contributed to the infringing activities of third parties in this 
District. 

26. This Court has jurisdiction over Defendant, because, whether directly or through 
an authorized agent, Defendant has been present in Washington, DC; transacted business in 
Washington, DC; caused tortious injury by an act or omission in Washington, DC; and/or caused 
tortious injury in Washington, DC by an act or omission outside Washington, DC. Defendant 
also regularly does and/or solicits business; engages in other persistent courses of conduct; 
and/or derives substantial revenue from goods used or consumed, or services rendered, in 
Washington, DC. 

FACTS 

Creation of the Standards for Educational and Psychological Testing 

27. In 1954, Plaintiff APA prepared and published the “Technical Recommendations 
for Psychological Tests and Diagnostic Techniques.” In 1955, Plaintiffs AERA and NCME 
prepared and published a companion document entitled, “Technical Recommendations for 
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Achievement Tests.” Subsequently, a joint committee of the three organizations modified, 
revised and consolidated the two documents into the first Joint Standards. Beginning with the 
1966 revision, the three organizations (AERA, APA and NCME - collectively, the “Plaintiffs”) 
collaborated in developing the “Joint Standards” (or simply, the “Standards”). Each subsequent 
revision of the Standards has been careful to cite the previous Standards and note that it is a 
revision and update of that document. 

28. Beginning in the mid-1950s, Plaintiffs formed and periodically reconstituted a 
committee of experts in psychological and educational assessment, charged with the initial 
development of the Technical Recommendations and then each subsequent revision of the 
(renamed) Standards. These committees were formed by the Plaintiffs’ Presidents (or their 
designees), who would meet and jointly agree on the membership. Often a chair or co-chairs of 
these committees were selected by joint agreement. Beginning with the 1966 version of the 
Standards, this committee became referred to as the “Joint Committee.” 

29. Many different fields of endeavor rely on assessments. Plaintiffs have ensured 
that the range of these fields of endeavor is represented in the Joint Committee’s membership - 
e.g., admissions, achievement, clinical counseling, educational, licensing-credentialing, 
employment, policy, and program evaluation. Similarly, the Joint Committee’s members 
represent expertise across major functional assessment areas - e.g., validity, equating, reliability, 
test development, scoring, reporting, interpretation, large scale interpolation and cognitive 
behavioral therapy. 

30. From the time of their initial creation to the present, the preparation and periodic 
revisions to the Standards entail intensive labor and considerable cross-disciplinary expertise. 
Each time the Standards are revised, Plaintiffs select and arrange for meetings of the leading 
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authorities in psychological and educational assessments. During these meetings, certain 
Standards are combined, pared down and/or augmented, others are deleted altogether, and some 
are created as whole new individual Standards. The most recent version of the Standards is 
nearly 200 pages, and took several years to complete. 

Why the Standards Were Created 

31. The Standards originally were created to improve professional practice in testing 
and assessment. The Standards can be used to guide the sound and ethical use of tests, and also 
to evaluate the quality of tests and testing practices. 

32. The Standards are intended to guide test developers and test users. Test user 
Standards refer to those that help decide how to choose certain tests, interpret scores, or make 
decisions based on tests results. Test users include clinical or industrial psychologists, research 
directors, school psychologists, counselors, employment supervisors, teachers, and various 
administrators who select or interpret tests for their organizations. The intended audience of the 
Standards is broad and cuts across audiences with varying backgrounds and different training. 
The Standards are not simply intended for members of the Plaintiffs that are the sponsoring 
organizations of the Standards. 

33. An essential component of responsible professional practice is maintaining 
technical competence. Many professional associations also have developed standards and 
principles of technical practice in assessment. The Standards have been widely cited to address 
technical, professional, and operational norms for all forms of assessments that are professionally 
developed and used in a variety of settings. 

34. The Standards are designed to apply to professional test developers, sponsors, 
publishers, and users by providing criteria for the evaluation of tests, testing practices, and the 
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effects of test use. The Standards have been used to develop testing guidelines for such activities 
as college admissions, personnel selection, test translations, test user qualifications, and 
computer-based testing. 

35. The Standards were not created in response to an expressed governmental or 
regulatory need, nor were they prepared in response to any legislative action or judicial decision. 
However, the Standards have been cited in judicial decisions related to the proper use and 
evidence for assessment, as well as by state and federal legislators. 

The Benefits that the Standards Bring to the Relevant Public and Constituencies 

36. The Standards’ primary purposes are to provide criteria for evaluating tests and 
testing practices, and to encourage test developers, sponsors, publishers, and users to adopt the 
Standards. There is no requirement for members of the professional associations or testing 
organizations and users to do so. There is no mechanism to enforce compliance with the 
Standards on the part of the test developer or test user. The Standards, moreover, do not attempt 
to provide psychometric answers to policy or legal questions. 

37. On the other hand, the Standards apply broadly to a wide range of standardized 
ins truments and procedures that sample an individual’s behavior, including tests, assessments, 
inventories, scales, and other testing vehicles. The Standards apply equally to standardized 
multiple-choice tests, performance assessments (including tests comprised of only open-ended 
essays), and hands-on assessments or simulations. The main exceptions are that the Standards do 
not apply to unstandardized questionnaires ( e.g ., unstructured behavioral checklists or 
observational forms), teacher-made tests, and subjective decision processes (e.g., a teacher’s 
evaluation of students’ classroom participation over the course of a semester). 
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Promotion and Distribution of the Standards, 

and Income Earned by Plaintiffs from Sales of the Standards 

38. Plaintiffs promote and sell copies of the Standards via their websites, at annual 
meetings, in public listings to students, and to educational institution faculty. Advertisements 
promoting the Standards have appeared in meeting brochures, in scholarly journals, and in the 
hallways at professional meetings. None of the Plaintiff organizations has solicited any 
government agency to incorporate the Standards into the Code of Federal Regulations or other 
rules of Federal or State agencies. 

39. As the designated publisher of the Standards, AERA sometimes provides 
promotional complementary print copies to students or professors. To date, the Standards have 
never been posted by any of the three Plaintiffs, or with the permission of the three Plaintiffs, on 
any publicly accessible website. All print copies of the Standards bear a copyright notice. 
Except for a few complementary print copies, the Standards are not digitized or given away for 
free; and certainly they are not made available to the public by any of the three organizations for 
anyone to copy free of charge. 

40. The Standards are sold at retail prices ranging from $35.00 to $40.00 per copy. 
Income generated from sales of the 1999 Standards, on average, has been approximately in 
excess of $127,000 per year. 

Plaintiffs’ Ownership of the Copyright in the Standards 

41. The Standards comprise an original work of authorship, entitled to protection 
under the U.S. Copyright Act, 17 U.S.C. §§ 101, et seq. 

42. The Plaintiffs are joint owners of the copyright in and to the Standards. 

43. The Standards have been registered with the U.S. Register of Copyrights under 
Registration Number TX 5-100-196, having an effective date of December 8, 1999. See Exhibit 
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A attached. 

44. A Supplementary copyright registration for the Standards has been issued by the 

U.S. Register of Copyrights under Supplementary Registration Number TX 6-434-609, having 

an effective date of February 25, 2014. See Exhibit B attached. 

Defendant’s Copying, and Enabling Others to 
Copy, the Standards without Plaintiffs’ Permission 

45. On its website, 

https://law.resource.org/pub/us/cfr/ibr/001/aera.standards.1999.pdf. Defendant has published in 
its entirety Plaintiffs’ 1999 Standards. 

46. Defendant published Plaintiffs’ 1999 Standards to its website, 
https://law.resource.org/pub/us/cfr/ibr/001/aera.standards.1999.pdf, without the permission or 
authorization of any of the Plaintiff organizations. 

47. In front of the unauthorized copy of the 1999 Standards that Defendant published 
to its https://law.resource.org/pub/us/cff/ibr/001/aera.standards.1999.pdf website, Defendant 
placed a cover sheet or “Certificate,” falsely implying that the publication of Plaintiffs’ 
Standards to Defendants’ website was somehow authorized or sanctioned by U.S. law. 

48. Defendant’s activities have encouraged others to publish the Standards without 
Plaintiffs’ permission or authorization. See, e.g., the posting of Plaintiffs’ 1999 Standards to the 
Internet Archive at https://archive.org/details/gov.law.aera.standards.1999 . The posting of 
Plaintiffs’ 1999 Standards to the Internet Archive is in the exact same format, and uses the same 
cover sheet or “Certificate” employed by Defendant in its publication of the Plaintiffs’ Standards 
to Defendant’s website. 

49. Plaintiff AERA requested in writing that Defendant remove the Standards from its 
online postings. Defendant refused. 
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FIRST CAUSE OF ACTION 

(Copyright Infringement) 

50. Plaintiffs repeat and re-allege paragraphs 1 through 49 of the Complaint. 

51. Defendant’s activities - publishing Plaintiffs’ Standards to the Internet without 
Plaintiffs’ permission or authorization - violate Plaintiffs’ exclusive rights under the U.S. 
Copyright Act, 17 U.S.C. § 106. 

52. Defendant’s activities - publishing Plaintiffs’ Standards to the Internet without 
Plaintiffs’ permission or authorization - constitute infringement of Plaintiffs’ copyright in the 
Standards. 

53. Defendant has knowingly, willfully and deliberately infringed Plaintiffs’ 
copyright in the Standards, and continues to do so. 

54. Plaintiffs have suffered and continue to suffer damages as a result of Defendant’s 
actions, in an amount to be proven at trial. 

55. During the period in which Defendant is infringing, Plaintiffs have no adequate 
remedy at law for the injuries that continue to be inflicted by Defendant. 

SECOND CAUSE OF ACTION 
(Contributory Copyright Infringement) 

56. Plaintiffs repeat and re-allege paragraphs 1 through 55 of the Complaint. 

57. Upon information and belief, in addition to Defendant’s direct infringement of 
Plaintiffs’ copyright in the Standards, other persons and/or entities have directly infringed 
Plaintiffs’ copyright in the Standards. 

58. Upon information and belief, Defendant has known, and/or with the exercise of 
reasonable diligence should have known, that other persons and/or entities have directly 
infringed Plaintiffs’ copyright in the Standards. 
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59. Upon information and belief, Defendant has substantially participated in other 
persons’ and/or entities’ direct infringement of Plaintiffs’ copyright in the Standards. 

60. Defendant has contributorily infringed Plaintiffs’ copyright in the Standards by 
intentionally inducing and/or encouraging others to directly infringe upon same. 

61. Defendant has knowingly, willfully and deliberately contributorily infringed 
Plaintiffs’ copyright in the Standards, and continues to do so. 

62. Plaintiffs have suffered and continue to suffer damages as a result of Defendant’s 
actions, in an amount to be proven at trial. 

63. During the period in which Defendant’s activities continue, Plaintiffs have no 
adequate remedy at law for the injuries continued to be inflicted by Defendant. 

RELIEF REQUESTED 

WHEREFORE, Plaintiffs, AERA, APA and NCME, request the following relief: 

A. Judgment in Plaintiffs’ favor and against Defendant; 

B. An injunction preliminarily and permanently enjoining and restraining Defendant, 
including its officers, directors, agents, employees, representatives, and all persons acting in 
concert with Defendant to (i) remove all infringing copies of Plaintiffs’ Standards from websites 
and any other locations under Defendant’s dominion and control; (ii) cease providing any 
members of the public with access to Plaintiffs’ Standards unless it is with Plaintiffs’ express 
written authorization and permission; and (iii) cease engaging in further acts constituting 
copyright infringement and/or contributory copyright infringement of Plaintiffs’ Standards, 
pursuant to 17 U.S.C. § 502. 

C. The impoundment and (where applicable) destruction of (i) all infringing copies 
of Plaintiffs’ Standards made or used by Defendant in violation of Plaintiffs’ copyright in the 
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Standards, (ii) all articles by means of which such infringing copies may be or have been 
reproduced, and (iii) all records documenting the manufacture, sale, or receipt of things involved 
in any such violation, provided that any records seized shall be taken into the custody of the 
court, pursuant to 17 U.S.C. §§ 503(a) and (b). 

D. An award of Plaintiffs’ costs and attorneys’ fees, pursuant to 17 U.S.C. § 505. 

E. Such other and further relief as the Court may deem just and proper. 


Respectfully submitted, 

OBLON, SPIVAK, MCCLELLAND, 
MAIER & NEUSTADT,LLP 


By: /$/ Jonathan Hudis _ 

Jonathan Hudis (DC Bar # 418872) 

Kathleen Cooney-Porter (DC Bar # 434526) 

1940 Duke Street 

Alexandria, VA 22314 

Tel. (703)413-3000 

Fax (703) 413-2220 

E-Mail ihudis@oblon.com 

E-Mail kcoonev-porter@oblon.com 

Attorneys for Plaintiffs 

AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC. 

AMERICAN PSYCHOLOGICAL 
ASSOCIATION, INC. 

NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC. 

jH:KCP:jh{i0259i79_i.Docx} Dated: May 23. 2014 
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USE 

ONLY 

~5 

“6 


b. Material Added to This Work Give a brief, general statement of the material that has been added to this work and in which copyright is claimed. ▼ 


betore completing 

Ns apace. 


MANUFA CTURERS AND LOCATIONS If this is a published work consisting preponderantly of nondramatie literary material in English, the law may 
require thrt the copies be manufactured in the United States or Canada for full protection. If so, the names of the manufacturers who performed certain 
processes, and the places where these processes were performed must be given. See instructions for details. 

Names of Manufacturers ▼ Placet of Manufacture ▼ 


7 


REPRODUCTION FOR USE OF BLIND OR PHYSICALLY HANDICAPPED INDIVIDUALS A signature on this form at space 10, and a 
check in one of the boxes here in space 8, constitutes a non-exdusive grant of permission to the Library of Congress to reproduce and distribute solely for the blind 
and physically handicapped and under the conditions and limitations prescribed by the regulations of the Copyright Office: (I) copies of the work Identified in space 
1 of this application in Braille (or similar tactile symbols); or (2) phonorecords embodying a fixation of a reading of that work; or (3) both. 

a O^Copies and Phonorecoidt b □ Copies Only e □ Phonorecords Only 

DEPOSIT ACCOUNT If the registration fee is to be charged to a Deposit Account established in the Copyright Office, give name and number of Account. 
Name ▼ Account Number ▼ 

American Educational Research Association_DAO 31143 


CORRESPONDENCE Give name and address to which correspondence about this application should be sent. Name/Address/Apt/CHy/State/ap T 

_ Camille S. Coy ____ 

1230 17th St., NW;■ 


Washington, DC 20036 


Aral Code A Telephone Number » Z02-2'2 3-9485 


CERTIFICATION* I, the undersigned, hereby certify that I am the 


of the work identified in this application and that the statements made 
by me in this application are correct to the best of my knowledge. 


I Q author * 

□ other copyright claimant 
□ owner of exdua^itghtd) 

» authorised agent of AERA _ 

Name oI author or otiar copyright claimant or owner of e«:Ul«e rightfs) A 


Typed or printed name and date ▼ If this is a published work, this date must be the Same as or lal 


8 

Sea instructions . 

9 

Bo aura to 
give your 
daytimt phone 
^number 

16 



J. S GOVERNMENT PRINTING OFFICE: 
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This Certificate issued under the seal of the Copyright 
Office in accordance with title 17, United States Code s 
attests that registration has been made for the work 
identified below The information-on this certificate has 
been made a part of the Copyright Office records. 

<XUc~~ 

Register of Copyrights, United .States of America 


Q FormCA 
~TX i-4S4-609 


ffmCTJVeOAT* OPaJmEMBTAIWIttaiffTTUnON 

X te suHi 


OOMOTM2ITm«WVWTr>«2 i UIJ»^VJ»^^roijOOTII«JC^l2BA»gJA2A¥^naifT»ttlAT10NMB«ar^ 


m * Kiglititiitn NambM Ol tS< Bifit ▼ 

TXB-10O.196 

1890 . * : , ’ i 

ol AotSarW ▼ 

American Educational Raaaareh AiiocfBon 

Nui(<i] ci (>7yrt«ht CUauBiW ▼ 

American Educational Raaaareh Association 


B 


tUo la Sub JU*Ufa*tfo» ▼ 

. UreHwdirmorPWqtpt^. 


Nairn of Author 


American Paychototfcal Association 


TWa book had three authors, but onty one waa Included In the orfflbiat filing. 


CowriQhtC 




n*m •** &p i« M a w 1 tm—daaT 
Tha addms* of the claimant Rstad haa chanjad to: 


1430 K Straat NW, 8uito 1200 
Washington, OC 20006 


•eiiiiMiipMr. 
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FORM CA RECEIVED 


ML 


EXAMINED BY 
CORRESPONDENCE 
raraMMCE TOTwi HHJuhsATtON ADOSDTO 

aii&aBignaBat jmam 


FOR 

COPYRIGHT 

OFFICII 


DO NOT Wltm ABOVK THIS UN 4. IF YOU MUD MORI BFAC8. UM AIgpAAATl CqNTMjAT10N SHEIT: 

Lina 2c was blink, but should have included the third author National Council on Measurement in Education 

Una 4 lifted one copyrltfit claimant: American Educational Reaeerch Association. Line 4 should have Hated 3 
copyright claimants aa IWad below: 


American Educational Research Association 
1430 K Street. NW 
Suite 1200 

Washington. DC 20005 

National Counci on Measurement in Education 
2424 American Lens 
Madison, Wi 53704 


American Psycholog leal Association 
790 First Street NE 
Washington, OC 20002-4242 


John NelWffc, 1430 K Street. NW 
Suita 1200 

Washington, DC 20005 


DtpaSI AssBSsSHd»i»|taeel«i(sSl»iBhsA«i|ed tssDspodt Acaoui«tMt*bli«h»dtndisCopyrl^lOtllcsilv*ai 



HjmSwrtUiR 

■am— 


enWjcaSi 
■ msasdia 

| John NoiMrfc, American Educational Raaeeroh Association 


todST" 

• 1430 K Street NW, Suita 1200 

tggSawmes 

sadrses: 

Washington. DC 2Q005 

BE 

SgggSSSBSS^^ P———-——--- 
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UNITED STATES DISTRICT COURT 
FOR THE DISTRICT OF COLUMBIA 


AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC. 

1430 K Street, NW 
Suite 1200 

Washington, DC 20005, 

AMERICAN PSYCHOLOGICAL 
ASSOCIATION, INC. 

750 First Street NE 
Washington, DC 20002-4242, and 

NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC. 
2424 American Lane 
Madison, WI 53704, 

Plaintiffs, 


PUBLIC.RESOURCE.ORG, INC. 
1005 Gravenstein Highway North 
Sebastopol, CA 95472, 

Defendant. 


) 

) 

) 

) 

) 

) 

) Civil Action No._ 

) 

) PLAINTIFFS’ DISCLOSURE 
) OF CORPORATE AFFILIATIONS 

) AND FINANCIAL INTERESTS 

) 

) 

) 

) 

) 

) 

) 

) 

) 

) 

) 

) 

) 

) 

) 


I, the undersigned, counsel of record for Plaintiffs, the American Educational Research 
Association, Inc., the American Psychological Association, Inc. and the National Council on 
Measurement in Education, Inc., certify that to the best of my knowledge and belief, none of the 
Plaintiffs have companies, subsidiaries or affiliates which have any outstanding securities in the 
hands of the public. 
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These representations are made in order that judges of this Court may determine the need 
for recusal. 


Respectfully submitted, 

OBLON, SPIVAK, MCCLELLAND, 
MAIER & NEUSTADT,LLP 


By: s/ Jonathan Hudis _ 

Jonathan Hudis (DC Bar # 418872) 

Kathleen Cooney-Porter (DC Bar # 434526) 

1940 Duke Street 

Alexandria, VA 22314 

Tel. (703)413-3000 

Fax (703) 413-2220 

E-Mail ihudis@oblon.com 

E-Mail kcooney-porter@oblon.com 

Attorneys for Plaintiffs 

AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC. 

AMERICAN PSYCHOLOGICAL 
ASSOCIATION, INC. 

NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC. 

JH:KCP:jh {10259444J.DOCX} Dated: May 23, 2014 
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CIVIL COVER SHEET 

44 (Rev. 7/13 PC) 


I. (a) PLAINTIFFS 

American Educational Research Association, Inc.; 
American Psychological Association, Inc.; 

National Council on Measurement in Education, Inc. 

(b) COUNTY OF RESIDENCE OF FIRST USTED PLAINTIFF 11001 _ 


(EXCEPT IN l .S. PLAINTIFF CASES) 


DEFENDANTS 

Public.Resource.Org, Inc. 

1005 Gravenstein Highway North 
Sebastapol, CA 95472 

COUNTY OF RESIDENCE OF FIRST USTED DEFENDANT _8 


<*■ LAND INVOLVED 


(c* ATTORNEYS (FIRM NAME, .ADDRESS, AND TELEPHONE NUMBER) 

Jonathan Hudis, Kathleen Cooney-Porter 
Obion, Spivak, McClelland, Maier & Neustadt L.L.P. 
1940 Duke Street 

Alexandria, VA 22314 703-413-3000 


ATTORNEYS (IF KNOWN) 


II BASIS OF JURISDICTION 

(PLACE AN x IN ONE BOX ONLY) 

O I U S Government (•) 3 Federal Question 

Plaintiff (US. Government Not a Party) 


O 2 U S. Government 
Defendant 


O 4 Diversity 

(Indicate Ciuzenship of 
Parties in item III) 


III. CITIZENSHIP OF PRIM IP \l, PARTIES (PLACE an x is one box for 


Citizen of this State 

PTF 

©1 

nn 

Oi 

Incorporated or Principal Place 

PTF 

04 

DFT 

04 

Citizen of Another State 

02 

©2 

of Business in This State 

Incorporated and Principal 

Os 

Os 

Citizen or Subject of a 
Foreign Country 

03 

03 

Place of Business in Another Stale 

Foreign Nation O 6 

06 


IV. CASE ASSIGNMENT AND NATURE OF SUIT 

(Place an X in one category, A-N, that best rep resents your Cause of Action and one in a co 


-ponding Nature of Suit) 


O A. Antitrust 

I 1 410 Antitrust 


O B. Personal Injury/ 
Malpractice 

I 1310 Airplane 
I 1 315 Airplane Product Liability 
I 1 320 Assault, Libel & Slander 
I 1 330 Federal Employers Liability 
I 1 340 Marine 

I 1345 Marine Product Liability 

□ 350 Motor Vehicle 

I 1 355 Motor Vehicle Product Liability 
I 1 360 Other Personal Injury 
I 1 362 Medical Malpractice 

□ 365 Product Liability 

I 1 367 Health Care/Pharmaceutical 

Personal Injury Product Liability 
I 1 368 Asbestos Product Liability 


O C. Administrative Agency 
Review 

I I 151 Medicare Acl 


Soc ial Security 
I I 861 HJA (139511) 

I I 862 Black Lung (923) 

I I 863 DIWCVDIWW (405(g)) 

□ 864 SSID Title XVI 
I I 865 RSI (405(g)) 

Olher Statutes 
I I 891 Agricultural Acts 

I 1 893 Environmental Matters 

I 1 890 Other Statutory Actions (If 
Administrative Agency is 
Involved) 


O D. Temporary Restraining 
Order/Preliminary 
Injunction 


"(If Antitrust, then A governs)* 


OR O 

F. Pro Se General Civil 

Bankruptcy 

Forfeiturc/Penaltv 

I_|625 Drug Related Seizure of 

□ 422 Appeal 27 I SC 158 

1 1423 Withdrawal 28 CSC 157 

Prisoner Petitions 

1 1535 Death Penalty 

Property 21 CSC 881 

1 1690 Other 

1_154(1 Mandamus & Other 

Other Statutes 

□ 550 Civil Rights 

1 1375 False Claims Act 

L_J 555 Prison Conditions 

1_1560 Civil Detainee - Conditions 

1 1400 State Reapportioamcnt 

1 1430 Banks & Banking 

of Confinement 

1 1450 Commerce/ICC 

Rates/etc. 

Property Rights 

1 X 1820 Copyrights 

1 1460 Deportation 

1 1462 Naturalization 

1 1830 Patent 

Application 

□ 840 Trademark 

1 1465 Other Immigration 

Actions 

Federal Tax Suits 

□ 470 Racketeer Influenced 

1 1870 Taxes (US plaintiff or 

defendant) 

1 187| IRS-Third Party 26 CSC 7609 

& Corrupt Organization 


© E. General Civil (Other) 


_210 UndO 

I 1220 Foreclosure 

I 1 230 Rent, Lease & Ejectment 

I 1 240 Torts to Und 

I 1 245 Tort Product Liability 
I 1 290 All Other Real Property 


_370 Other Fraud 

I 1 371 Truth in Lending 
I 1 380 Other Personal Properly 
Damage 

I 1385 Property Damage 
Product Liability 


I 1480 Consumer ( redit 

I 1490 Cable/Satellite TV 
I I 8S0 Securities/Commodities/ 
Exchange 

I 1 896 Arbitration 
I 1 899 Administrative Procedure 
Act/Review or Appeal of 
Agency Decision 

I 1 950 Constitutionality of State 
Statutes 

I 1890 Other Statutory Actions 

(if not administrative agency 
review or Privacy Act) 
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O G. Habeas Corpus/ 
2255 

I I 530 Habeas Corpus - General 
I I SIP Motion/Vacate Sentence 
I 1 463 Habeas Corpus - Alien 


O H. Employment 
Discrimination 

I | 442 Civil Rights - Employment 
(criteria: race, gcnder/sei, 
national origin, 
discrimination, disability, age, 
religion, retaliation) 

*(If pro se, select this deck)* 


O I. FOIA/Privacy Act 

I 1 895 Freedom of Information A 
I 1 890 Other Statutory Actions 
(if Privacy Act) 


*(lf pro se, select this deckl* 


O J. Student Loan 

I 1 152 Recovery of Defaulted 
Student Loan 
(excluding veterans) 


O K. Labor/ERISA 

(non-employment) 

I 1 710 Fair labor Standards Act 
I 1 720 Labor/Mgmt Relations 
i 1 740 Labor Railway Act 
I 1 751 Family and Medical 

I 1 790 Other Labor Litigation 
I 1 791 Empl. Ret Inc. Security Act 


C 1- Other Civil Rights 
(non-employment) 

I 1 441 Voting (if not Voting Rights 
Act) 

I 1 443 Housing/Accommodations 
I 1 440 Other Civil Rights 
I 1 445 Americans w/Disabilities - 

Employment 

I 1 446 Americans w/Disabilities - 

Other 

I 1 448 Education 


O M. Contract 

I 1 110 Insurance 
□ 120 Marine 
I 1 130 Miller Act 
I 1 140 Negotiable Instrument 
I 1 150 Recovery of Overpayment 
& Enforcement of 
Judgment 

I 1 153 Recovery of Overpayment 
of Veteran’s Benefits 
I 1 160 Stockholder’s Suits 
I 1 190 Other Contracts 
I 1 195 Contract Product Liability 

I 1 196 Franchise 


O N. Three-Judge 
Court 


I I 441 Civil Rights - N oting 
(if Voting Rights Act) 


© I Original O 2 Removed O 3 Remanded from O 4 Reinstated or O * transferred from O 6 Multi-disl 
Proceeding from State Appellate Court Reopened another district 

Court (specify) 


07 Appeal to 

District Judge 
from Mag. Judge 


VI. CAUSE OF ACTION (CITE THE ll.S. CIVIL STATUTE UNDER WHICH YOU ARE FILING AND WRITE A BRIEF STATEMENT OF CAI SE.) 

Copyright Infringement and Contributory Copyright Infringement; 17 U.S.C §§ 101 et seq. 


Check YES on ly if dem anded in complaint 

YES □ NO 


VIII. RELATED CASE(S) 
IF ANY 


NO If yes, please complete related case fc 



INSTRUCTIONS FOR COMPLETING CIVIL COVER SHEET JS-44 
Authority for Civil Cover Sheet 


The JS-44 civil cover sheet and the information contained herein neither replaces nor supplements the filings and services of pleadings or other papers as required 
by law, except as provided by local rules of court. This form, approved by the Judicial Conference of the United States in September 1974, is required for the use of the 
Clerk of Court for the purpose of initiating the civil docket sheet Consequently, a civil cover sheet is submitted to the Cleric of Court for each civil complaint filed. 
Listed below are bps for completing the civil cover sheet These tips coincide with the Roman Numerals on the cover sheet. 

I. COUNTY OF RESIDENCE OF FIRST LISTED PLAINTIFF/DEFENDANT (b) County of residence Use 11001 to indicate plaintiff'll'resident 

of Washington, DC, 88888 if plaintiff is resident of United Slates but not Washington. DC, and 99999 if plaintiff is outside the United States. 

HI. CITIZENSHIP OF PRINCIPAL PARTIES: This section is completed only if diversity of citizenship was selected as the Basis of Jurisdiction 
under Section II. 

IV’. CASE ASSIGNMENT AND NATURE OF SUIT The assignment of a judge to your case will depend on the category you select that best 
represents the primary cause of action found in your complaint You may select only ons category You must also select one corresponding 
nature of suit found under the category of the case 

VL CAUSE OF ACTION: Cite the US. Civil Statute under which you are filing and write a brief statement of the primary cause. 

VIII. RELATED C.ASE(S), IF ANY If you indicated that there is a related case, you must complete a related case form, which may he obtained from 
the Clerk’s Office 


Because of the need for accurate and complete information, you should ensure the accuracy of the information provided prior to signing the form 
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AO 440 (Rev. 12/09; DC 03/10) Summons in a Civil Action 


United States District Court 

for the 

District of Columbia 


American Educational Research Association, Inc 
et al. 

Plaintiff 

V. 

Public.Resource.Org, Inc. 


Defendant 


SUMMONS IN A CIVIL ACTION 

To: (Defendant’s name and address) Public. Resource. Org, Inc. 

1005 Gravenstein Highway North 
Sebastapol, CA 95472 


) 

) 

) 

) Civil Action No. 
) 

) 

) 


A lawsuit has been filed against you. 

Within 21 days after service of this summons on you (not counting the day you received it) — or 60 days if you 
are the United States or a United States agency, or an officer or employee of the United States described in Fed. R. Civ. 
P. 12 (a)(2) or (3) — you must serve on the plaintiff an answer to the attached complaint or a motion under Rule 12 of 
the Federal Rules of Civil Procedure. The answer or motion must be served on the plaintiff or plaintiffs attorney, 
whose name and address are: Jonathan Hudis, Kathleen Cooney-Porter 

Obion, Spivak, McClelland, Maier & Neustadt L.L.P. 

1940 Duke Street 
Alexandria, VA 22314 
703-413-3000 


If you fail to respond, judgment by default will be entered against you for the relief demanded in the complaint. 
You also must file your answer or motion with the court. 


ANGELA D. CAESAR, CLERK OF COURT 


Date: _ _ 

Signature of Clerk or Deputy Clerk 
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AO 440 (Rev. 12/09; DC 03/10) Summons in a Civil Action (Page 2) 


Civil Action No. 


PROOF OF SERVICE 

(This section should not be filed with the court unless required by Fed. R. Civ. P. 4 (l)) 

This summons for (name of individual and title, if any) 
was received by me on (date) 

□ I personally served the summons on the individual at (place) 

on (date) ; or 

□ I left the summons at the individual’s residence or usual place of abode with (name) 

, a person of suitable age and discretion who resides there, 
on (date) , and mailed a copy to the individual’s last known address; or 

□ I served the summons on (name of individual) , who is 

designated by law to accept service of process on behalf of (name of organization) 

on (date) ; or 

□ I returned the summons unexecuted because ; or 

□ Other (specify): 


My fees are $ for travel and $ for services, for a total of $ 0.00 


I declare under penalty of perjury that this in formation is true. 


Date: _ _ 

Server's signature 


Printed name and title 


Server’s address 


Additional information regarding attempted service, etc: 
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TO: 


Register of Copyrights 

REPORT ON THE 

Copyright Office 

FILING OR DETERMINATION OF AN 

Library of Congress 

ACTION OR APPEAL 

Washington, D.C. 20559 

REGARDING A COPYRIGHT 


In compliance with the provisions of 17 U.S.C. 508, you are hereby advised that a court action or appeal has been filed 
on the following copyright(s): 


0 ACTION □ APPEAL 

COURT NAME AND LOCATION 

United States District Court for The District of Columbia 

333 Constitution Ave. N.W. 

Washington, D.C. 20001 

DOCKET NO. 

DATE FILED 

5/23/2014 

PLAINTIFF 

American Educational Research Association, Inc.; 

American Psychological Association, Inc.; 

National Council on Measurement in Education, Inc. 

DEFENDANT 

Public.Resource.Org, Inc. 

1005 Gravenstein Highway North 

Sebastapol, CA 95472 

COPYRIGHT 
REGISTRATION NO. 

TITLE OF WORK 

AUTHOR OR WORK 

1 TX 5-100-196 

Standards for Educational and Psychological Testing 

AERA; APA; NCME 

2 



3 



4 



5 




In the above-entitled case, the following copyright(s) have been included: 


DATE INCLUDED 

INCLUDED BY 

□ Amendment □ Answer □ Cross Bill □ Other Pleading 

COPYRIGHT 
REGISTRATION NO. 

TITLE OF WORK 

AUTHOR OF WORK 

1 



2 



3 




In the above-entitled case, a final decision was rendered on the date entered below. A copy of the order or judgment 
together with the written opinion, if any, of the court is attached. 


COPY ATTACHED 


WRITTEN OPINION ATTACHED 

DATE RENDERED 

□ Order 

I~1 Judgment 

□ Yes □ No 



(BY) DEPUTY CLERK 


DISTRIBUTION: 


nt of an appeal, forward copy to Appellate Court 
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IN THE UNITED STATES DISTRICT COURT 
FOR THE DISTRICT OF COLUMBIA 


AMERICAN EDUCATIONAL RESEARCH ) 

ASSOCIATION, INC., AMERICAN ) 

PSYCHOLOGICAL ASSOCIATION, INC., ) 

and NATIONAL COUNCIL ON ) 

MEASUREMENT IN EDUCATION, INC., ) 

) 

Plaintiffs/Counterclaim-Defendants, ) 

) 

v. ) 

) 

PUBLIC.RESOURCE.ORG, INC., ) 

) 

Defendant/Counterclaim-Plaintiff. ) 

_ ) 


Civil Action No. l:14-cv-00857-CRC 

PLAINTIFFS’ REPLY AND 
AFFIRMATIVE DEFENSES TO 
DEFENDANT’S COUNTERCLAIM 
FOR DECLARATORY RELIEF 


Plaintiffs/Counterclaim-Defendants, American Educational Research Association, Inc. 
(“AERA”), American Psychological Association, Inc. (“APA”), and National Council on 
Education Measurement in Education, Inc. (“NCME”) (collectively, “Plaintiffs”), as and for their 
Reply and Affirmative Defenses to the Counterclaim for Declaratory Relief of 
Defendant/Counterclaim-Plaintiff, Public.Resource.Org, Inc. (“Public Resource” or 
“Defendant”), responds as follows: 

Definitions 

A. As used herein, the term “1999 Standards” shall mean the 1999 edition of the 
Standards for Educational and Psychological Testing. 

B. As used herein, the term “1985 Standards” shall mean the 1985 edition of the 
Standards for Educational and Psychological Testing. 

Responses to Counterclaim 

Introduction 

1. Plaintiffs admit that Public Resource has posted numerous copyrighted standards 
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and codes on websites under its dominion and control. Plaintiffs otherwise lack information 
sufficient to form a belief as to the remaining allegations contained in paragraph 1 of Public 
Resource’s Counterclaim, and therefore deny same leaving Defendant to its proofs. 

2. Plaintiffs generally admit that federal, state and local governments within the 
United States promulgate regulations. Plaintiffs otherwise do not understand what Public 
Resource means by the term “fundamental component,” and therefore deny the remaining 
allegations contained in paragraph 2 of Public Resource’s Counterclaim, leaving Defendant to its 
proofs. 

3. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 3 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

4. Plaintiffs deny the allegations contained in paragraph 4 of Public Resource’s 
Counterclaim. 

5. Plaintiffs admit that the Code of Federal Regulations cited by Public Resource in 
paragraph 5 of Public Resource’s Counterclaim state that the 1999 Standards are incorporated by 
reference within those Code provisions. Plaintiffs otherwise deny the allegations contained in 
paragraph 5 of Public Resource’s Counterclaim. 

6. Plaintiffs ad mi t that the state code or rule provisions cited by Public Resource in 
paragraph 6 of Public Resource’s Counterclaim state that the 1999 Standards are incorporated by 
reference within those code provisions. Plaintiffs otherwise deny the allegations contained in 
paragraph 5 of Public Resource’s Counterclaim. 

7. Plaintiffs admit the allegations contained in paragraph 7 of Public Resource’s 
Counterclaim. 


- 2 - 
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8. Plaintiffs admit that they own the copyright in the 1999 Standards. Plaintiffs 
otherwise do not understand what Public Resource means by the term “privileged relationship,” 
and therefore deny the remaining allegations contained in paragraph 8 of Public Resource’s 
Counterclaim, leaving Defendant to its proofs. 

9. Plaintiffs admit that they own the copyright in the 1999 Standards. Plaintiffs 
otherwise do not understand what Public Resource means by the term “privileged relationship,” 
and therefore deny the remaining allegations contained in paragraph 9 of Public Resource’s 
Counterclaim, leaving Defendant to its proofs. 

10. Plaintiffs admit that, as owner of the copyright in the 1999 Standards, they are 
endowed with the benefits and privileges of copyright protection to that work under the U.S. 
Copyright Act, 17 U.S.C. §§ 101 et seq. Plaintiffs otherwise do not understand what Public 
Resource means by the term “lawful opportunity,” and therefore deny the remaining allegations 
contained in paragraph 10 of Public Resource’s Counterclaim, leaving Defendant to its proofs. 

11. Plaintiffs admit that, as owner of the copyright in the 1999 Standards, they are 
endowed with the benefits and privileges of copyright protection to that work under the U.S. 
Copyright Act, 17 U.S.C. §§ 101 et seq. Plaintiffs otherwise do not understand what Public 
Resource means by the term “lawful opportunity,” and therefore deny the remaining allegations 
contained in paragraph 11 of Public Resource’s Counterclaim, leaving Defendant to its proofs. 

12. Plaintiffs admit that, as owner of the copyright in the 1999 Standards, they are 
endowed with the benefits and privileges of copyright protection to that work under the U.S. 
Copyright Act, 17 U.S.C. §§ 101 et seq. Plaintiffs otherwise do not understand what Public 
Resource means by the term “lawful opportunity,” and therefore deny the remaining allegations 
contained in paragraph 12 of Public Resource’s Counterclaim, leaving Defendant to its proofs. 

-3- 
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13. Plaintiffs admit that, as owner of the copyright in the 1999 Standards, they are 
endowed with the benefits and privileges of copyright protection to that work under the U.S. 
Copyright Act, 17 U.S.C. §§ 101 et seq. Plaintiffs otherwise do not understand what Public 
Resource means by the terms “lawful opportunity,” “certain activities,” and “gatekeepers,” and 
therefore deny the remaining allegations contained in paragraph 13 of Public Resource’s 
Counterclaim, leaving Defendant to its proofs. 

14. Plaintiffs admit that, as owner of the copyright in the 1999 Standards, they are 
endowed with the benefits and privileges of copyright protection to that work under the U.S. 
Copyright Act, 17 U.S.C. §§ 101 et seq. - including the right to set the conditions under which 
the public may use the 1999 Standards consistent with the rights afforded by the U.S. Copyright 
Act. Plaintiffs otherwise do not understand what Public Resource means by the terms “have 
access” and “gain certain rights,” and therefore deny the remaining allegations contained in 
paragraph 14 of Public Resource’s Counterclaim, leaving Defendant to its proofs. 

15. The allegations contained in paragraph 15 of Public Resource’s Counterclaim are 
legal conclusions to which no response is required. Plaintiffs otherwise do not understand what 
Public Resource means by the term “content of law” and therefore deny the remaining 
allegations contained in paragraph 15 of Public Resource’s Counterclaim, leaving Defendant to 
its proofs. By way of further answer, Plaintiffs provide the public with access to the 1999 
Standards, such that no action by Public Resource is required to provide the public with access to 
the 1999 Standards. 

16. The allegations contained in paragraph 16 of Public Resource’s Counterclaim are 
legal conclusions to which no response is required. Plaintiffs otherwise do not understand what 
Public Resource means by the term “content of law” and therefore deny the remaining 
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allegations contained in paragraph 16 of Public Resource’s Counterclaim, leaving Defendant to 
its proofs. By way of further answer, Plaintiffs provide the public with access to the 1999 
Standards, such that no action by Public Resource is required to provide the public with access to 
the 1999 Standards. 

17. The allegations contained in paragraph 17 of Public Resource’s Counterclaim are 
legal conclusions to which no response is required. By way of further answer, Plaintiffs admit 
that, as owner of the copyright in the 1999 Standards, they are endowed with the benefits and 
privileges of copyright protection to that work under the U.S. Copyright Act, 17 U.S.C. §§ 101 et 
seq. 

18. Plaintiffs admit the allegations contained in paragraph 18 of Public Resource’s 
Counterclaim. 

19. Plaintiffs admit that U.S. copyright law gives Plaintiffs the power to determine the 
conditions under which Public Resource and others may access, reproduce, publish, translate, 
reformat or annotate the Standards, or enable others to do so. Except as specifically admitted, 
Plaintiffs deny the allegations contained in paragraph 19 of Public Resource’s Counterclaim. 

Parties 

20. Upon information and belief, Plaintiffs admit the allegations contained in 
paragraph 20 of Public Resource’s Counterclaim. 

21. Plaintiffs admit the allegations contained in paragraph 21 of Public Resource’s 
Counterclaim. 

22. Plaintiffs admit the allegations contained in paragraph 22 of Public Resource’s 
Counterclaim. 

23. Plaintiffs admit the allegations contained in paragraph 23 of Public Resource’s 
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Counterclaim. 

Jurisdiction and Venue 

24. Plaintiffs admit that Public Resource’s Counterclaim purports to seek declaratory 
relief pursuant to 28 U.S.C. § 2201 and 17 U.S.C. §§ 101 et seq., and admits that the Court has 
subject matter jurisdiction over the Counterclaim. Except as specifically admitted, Plaintiffs 
deny the allegations contained in paragraph 24 of Public Resource’s Counterclaim. 

25. Plaintiffs admit that AERA and APA maintain offices in Washington, D.C., that 
all of the Plaintiffs transact business in this District, and that the Court has personal jurisdiction 
over each of the Plaintiffs. Except as specifically admitted, Plaintiffs deny the allegations 
contained in paragraph 25 of Public Resource’s Counterclaim. 

26. Plaintiffs admit that they have submitted to the Court’s jurisdiction over each of 
them for purposes of this case by filing their Complaint against Public Resource in this Court. 
Except as specifically admitted, Plaintiffs deny the allegations contained in paragraph 26 of 
Public Resource’s Counterclaim. 

27. Plaintiffs admit that venue is proper in this Court, that AERA and APA maintain 
offices in Washington, D.C., and that Plaintiffs filed their Complaint against Public Resource in 
this Court. Except as specifically admitted, Plaintiffs deny the allegations contained in 
paragraph 27 of Public Resource’s Counterclaim. 

Facts 

I. Public Resource’s Operations 

28. Plaintiffs admit that Public Resource is a California not-for-profit corporation. 
Plaintiffs otherwise lack information sufficient to form a belief as to the allegations contained in 
paragraph 28 of Public Resource’s Counterclaim, and therefore deny same leaving Defendant to 
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its proofs. 

29. Plaintiffs lack information sufficient to form a belief as to the allegations 
concerning Public Resource contained in paragraph 29 of Public Resource’s Counterclaim, and 
therefore deny same leaving Defendant to its proofs. Plaintiffs otherwise admit the allegations 
contained in paragraph 29 of Public Resource’s Counterclaim. 

30. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 30 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

31. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 31 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

32. Plaintiffs admit the allegations contained in paragraph 32 of Public Resource’s 
Counterclaim. 

33. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 33 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

34. Plaintiffs do not understand what Public Resource means by the term “hosts 
standards” and therefore deny the allegations contained in paragraph 34 of Public Resource’s 
Counterclaim, leaving Defendant to its proofs. 

35. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 35 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

36. Plaintiffs lack information sufficient to form a belief as to the allegations 
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contained in paragraph 36 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

37. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 37 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

38. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 38 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

39. Plaintiffs deny the allegations contained in paragraph 39 of Public Resource’s 
Counterclaim. 

40. Plaintiffs deny the allegations contained in paragraph 40 of Public Resource’s 
Counterclaim. 

41. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 41 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

42. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 42 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

43. Plaintiffs admit the allegations contained in paragraph 43 of Public Resource’s 
Counterclaim. 

44. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 44 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 
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II. Plaintiffs’ Operations 

45. Plaintiffs admit the allegations contained in paragraph 45 of Public Resource’s 
Counterclaim. 

46. Plaintiffs admit the allegations contained in paragraph 46 of Public Resource’s 
Counterclaim. 

47. Plaintiffs admit the allegations contained in paragraph 47 of Public Resource’s 
Counterclaim. 

48. Plaintiffs admit that their development of the 1999 Standards reduces duplication 
of effort for test writers. Except as specifically admitted, Plaintiffs deny the allegations 
contained in paragraph 48 of Public Resource’s Counterclaim. 

49. Plaintiffs admit the allegations contained in paragraph 49 of Public Resource’s 
Counterclaim. 

50. Plaintiffs admit the allegations contained in paragraph 50 of Public Resource’s 
Counterclaim. 

51. Plaintiffs admit the allegations contained in paragraph 51 of Public Resource’s 
Counterclaim. 

52. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 52 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

53. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 52 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

54. Plaintiffs admit that they update the Standards for Educational and Psychological 
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Testing periodically. Except as specifically admitted, Plaintiffs deny the allegations contained in 
paragraph 54 of Public Resource’s Counterclaim. 

55. Plaintiffs admit that Public Resource published the entirety of the 1999 Standards 
to Public Resource’s https://law.resource.org website. Except as specifically admitted, Plaintiffs 
deny the allegations contained in paragraph 55 of Public Resource’s Counterclaim. 

56. Plaintiffs admit that the U.S. Department of Education has referenced the 1999 
Standards in Title 34 of the U.S. Code of Federal Regulations. Except as specifically admitted, 
Plaintiffs deny the allegations contained in paragraph 56 of Public Resource’s Counterclaim. 

57. Plaintiffs admit the allegations contained in paragraph 57 of Public Resource’s 
Counterclaim. 

58. Plaintiffs do not understand what Public Resource means by the phrase “is not 
incorporated into federal regulations,” and therefore deny the allegations contained in paragraph 
58 of Public Resource’s Counterclaim, leaving Defendant to its proofs. 

59. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 59 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

60. Plaintiffs deny the allegations contained in paragraph 60 of Public Resource’s 
Counterclaim. 

61. Plaintiffs admit the allegations contained in paragraph 61 of Public Resource’s 
Counterclaim. 

62. Plaintiffs admit the allegations contained in paragraph 62 of Public Resource’s 
Counterclaim. 

63. Plaintiffs admit the allegations contained in paragraph 63 of Public Resource’s 
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Counterclaim. 

64. Plaintiff AERA admits the allegations contained in paragraph 64 of Public 
Resource’s Counterclaim. 

65. Plaintiff APA admits the allegations contained in paragraph 65 of Public 
Resource’s Counterclaim. 

66. Plaintiff NCME admits the allegations contained in paragraph 66 of Public 
Resource’s Counterclaim. 

67. Plaintiffs admit that the Federal Register Notice cited by Public Resource in 
paragraph 67 of Public Resource’s Counterclaim state that the 1985 Standards, as amended June 
2, 1989, were incorporated by reference into certain Sections of the Code of Federal Regulations. 
Plaintiffs otherwise deny the allegations contained in paragraph 67 of Public Resource’s 
Counterclaim. 

68. Plaintiffs deny the allegations contained in paragraph 68 of Public Resource’s 
Counterclaim. 

69. Plaintiff AERA admits that it was aware, prior to the filing of this action, that the 
1999 Standards were referenced by the U.S. Department of Education in Title 34 of the U.S. 
Code of Federal Regulations. Except as specifically admitted, Plaintiff AERA denies the 
allegations contained in paragraph 69 of Public Resource’s Counterclaim. 

70. Plaintiff APA admits that it was aware, prior to the filing of this action, that the 
1999 Standards were referenced by the U.S. Department of Education in Title 34 of the U.S. 
Code of Federal Regulations. Except as specifically admitted, Plaintiff APA denies the 
allegations contained in paragraph 70 of Public Resource’s Counterclaim. 

71. Plaintiff NCME admits that it was aware, prior to the filing of this action, that the 
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1999 Standards were referenced by the U.S. Department of Education in Title 34 of the U.S. 
Code of Federal Regulations. Except as specifically admitted, Plaintiff NCME denies the 
allegations contained in paragraph 69 of Public Resource’s Counterclaim. 

72. Plaintiff AERA admits the allegations contained in paragraph 72 of Public 
Resource’s Counterclaim, but denies that making such a request of any U.S. Government entity 
is an obligation of AERA in maintaining the enforceability of the copyright in the 1999 
Standards. 

73. Plaintiff APA admits the allegations contained in paragraph 73 of Public 
Resource’s Counterclaim, but denies that making such a request of any U.S. Government entity 
is an obligation of APA in maintaining the enforceability of the copyright in the 1999 Standards. 

74. Plaintiff NCME admits the allegations contained in paragraph 74 of Public 
Resource’s Counterclaim, but denies that making such a request of any U.S. Government entity 
is an obligation of NCME in maintaining the enforceability of the copyright in the 1999 
Standards. 

75. Plaintiff AERA admits that it never requested any U.S. Government entity for 
compensation if and when the Standards were referenced or mentioned in governmental 
regulations, but denies that making such a request of any U.S. Government entity is an obligation 
of AERA in maintaining the enforceability of the copyright in the 1999 Standards. Plaintiff 
AERA otherwise the allegations contained in paragraph 75 of Public Resource’s Counterclaim. 

76. Plaintiff APA admits that it never requested any U.S. Government entity for 
compensation if and when the Standards were referenced or mentioned in governmental 
regulations, but denies that making such a request of any U.S. Government entity is an obligation 
of APA in maintaining the enforceability of the copyright in the 1999 Standards. Plaintiff APA 
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otherwise the allegations contained in paragraph 76 of Public Resource’s Counterclaim. 

77. Plaintiff NCME admits that it never requested any U.S. Government entity for 
compensation if and when the Standards were referenced or mentioned in governmental 
regulations, but denies that making such a request of any U.S. Government entity is an obligation 
of NCME in maintaining the enforceability of the copyright in the 1999 Standards. Plaintiff 
NCME otherwise the allegations contained in paragraph 77 of Public Resource’s Counterclaim. 

78. Plaintiff AERA admits that it never protested to any U.S. Government entity if 
and when the Standards were referenced or mentioned in governmental regulations. Plaintiff 
AERA does not understand what Public Resource means by the phrase “incorporation of any 
edition ... into regulations,” and therefore denies the allegations contained in paragraph 78 of 
Public Resource’s Counterclaim as stated, leaving Defendant to its proofs. Plaintiff AERA 
generally denies that making a protest to any U.S. Government entity is an obligation of AERA 
in maintaining the enforceability of the copyright in the 1999 Standards. 

79. Plaintiff APA ad mi ts that it never protested to any U.S. Government entity if and 
when the Standards were referenced or mentioned in governmental regulations. Plaintiff APA 
does not understand what Public Resource means by the phrase “incorporation of any edition ... 
into regulations,” and therefore denies the allegations contained in paragraph 79 of Public 
Resource’s Counterclaim as stated, leaving Defendant to its proofs. Plaintiff APA generally 
denies that making a protest to any U.S. Government entity is an obligation of APA in 
maintaining the enforceability of the copyright in the 1999 Standards. 

80. Plaintiff NCME admits that it never protested to any U.S. Government entity if 
and when the Standards were referenced or mentioned in governmental regulations. Plaintiff 
NCME does not understand what Public Resource means by the phrase “incorporation of any 
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edition ... into regulations,” and therefore denies the allegations contained in paragraph 80 of 
Public Resource’s Counterclaim as stated, leaving Defendant to its proofs. Plaintiff NCME 
generally denies that making a protest to any U.S. Government entity is an obligation of NCME 
in maintaining the enforceability of the copyright in the 1999 Standards. 

81. Plaintiffs admit that, at one time, the website located at 
http://www.teststandards.org contained an article authored by Wayne Camara which stated that 
the Standards for Educational and Psychological Testing have been “referenced in law and cited 
in Supreme Court and other judicial decisions lending additional authority to the document.” 
However, the teststandards.org website was recently updated and no longer contains the Camara 
article containing the quoted text in question. 

III. Standards that the Law (Allegedly) Incorporates 

82. The allegations contained in paragraph 82 of Public Resource’s Counterclaim are 
legal conclusions to which no response is required. 

83. Plaintiffs admit that, at 69 Fed. Reg. 39913 at 39914, col. 1 (July 1, 2004), it 
states: “[t]o the greatest possible degree, the principles and guidelines developed under this goal 
must be compatible extensions of the Standards for Educational and Psychological Testing.” 
Except as specifically admitted, Plaintiffs deny the allegations contained in paragraph 83 of 
Public Resource’s Counterclaim. 

84. Plaintiffs deny the allegations contained in paragraph 84 of Public Resource’s 
Counterclaim. By way of further answer, the section of the Minn. Ad mi n Rules cited by Public 
Resource was repealed. 

85. Plaintiffs do not understand what Public Resource means by the phrase “[ojther 
regulations, funding opportunities, and Requests for Proposals” and therefore deny the 
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allegations contained in paragraph 85 of Public Resource’s Counterclaim, leaving Defendant to 
its proofs. 

86. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 86 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

87. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 87 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

88. Plaintiffs admit that public comments often are solicited during the process of 
drafting and adopting statutes and regulations. Plaintiffs otherwise lack information sufficient to 
form a belief as to the allegations contained in paragraph 88 of Public Resource’s Counterclaim, 
and therefore deny same leaving Defendant to its proofs. 

89. Plaintiffs do not understand what Public Resource means by “incorporation of a 
standard into law,” and therefore deny the allegations contained in paragraph 89 of Public 
Resource’s Counterclaim, leaving Defendant to its proofs. 

90. Plaintiffs do not understand what Public Resource means by “the incorporation of 
a standard,” and therefore deny the allegations contained in paragraph 90 of Public Resource’s 
Counterclaim, leaving Defendant to its proofs. 

91. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 91 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

92. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 92 of Public Resource’s Counterclaim, and therefore deny same leaving 
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Defendant to its proofs. 

93. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 93 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. Plaintiffs cannot purport to answer Public Resource’s allegations on 
behalf of the general public or which relate to some amorphous reference to unknown “laws and 
regulations.” 

94. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 94 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. Plaintiffs cannot purport to answer Public Resource’s allegations on 
behalf of the general public regarding their interest in bias (or lack thereof) in standardized 
testing. 

95. Plaintiffs deny the allegations contained in paragraph 95 of Public Resource’s 
Counterclaim. By way of further answer, Plaintiffs provide the public with access to the 1999 
Standards in other ways not mentioned by Public Resource in paragraph 95. 

96. Plaintiffs admit the allegations contained in paragraph 96 of Public Resource’s 
Counterclaim. 

97. Plaintiffs do not understand what Public Resource means by the phrase “provides 
access to the contents,” and therefore deny the allegations contained in paragraph 97 of Public 
Resource’s Counterclaim leaving Defendant to its proofs. By way of further answer, the 
unauthorized version of the 1999 Standards published to Public Resource’s website was not in 
electronically searchable format. 

98. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 98 of Public Resource’s Counterclaim, and therefore deny same leaving 
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Defendant to its proofs. 

99. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 99 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

100. Plaintiffs admit that, in 47 Fed. Reg. 34108 (Aug. 6, 1982), Sec. 51.1(c)(2), it 
states: incorporation by reference “[i]s not intended to detract from the legal or practical 
attributes of the system established by the Federal Register Act, the Administrative Procedure 
Act, the regulations of the Administrative Committee of the Federal Register, and the acts which 
require publication in the Federal Register.” Except as specifically admitted, Plaintiffs deny the 
allegations contained in paragraph 100 of Public Resource’s Counterclaim. 

101. Plaintiffs do not understand what Public Resource means by the phrase “charges 
fees for access,” and therefore deny the allegations contained in paragraph 101 of Public 
Resource’s Counterclaim, leaving Defendant to its proofs. By way of further answer, as 
publisher of record, Plaintiff AERA sells printed copies of the 1999 Standards at retail prices 
ranging from $35.95 to $49.95 per copy, and distributes the net income from these sales to the 
three copyright owner organizations (i.e., AERA, APA and NCME). 

102. Plaintiff AERA does not understand what Public Resource means by the phrase 
“charges fees for access,” and therefore deny the allegations contained in paragraph 102 of 
Public Resource’s Counterclaim, leaving Defendant to its proofs. By way of further answer, as 
publisher of record, Plaintiff AERA sells printed copies of the 1999 Standards at retail prices 
ranging from $35.95 to $49.95 per copy, and distributes the net income from these sales to the 
three copyright owner organizations (i.e., AERA, APA and NCME). 

103. Plaintiffs admit that, Amazon.com is a re-seller and renter of printed copies of the 
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1999 Standards, for which Amazon.com sets its own prices. Plaintiffs otherwise lack 
information sufficient to form a belief as to the allegations contained in paragraph 103 of Public 
Resource’s Counterclaim, leaving Defendant to its proofs. 

104. Plaintiffs deny the allegations contained in paragraph 104 of Public Resource’s 
Counterclaim. 

105. Plaintiffs deny the allegations contained in paragraph 105 of Public Resource’s 
Counterclaim. 

106. Plaintiffs deny the allegations contained in paragraph 106 of Public Resource’s 
Counterclaim. 

107. Plaintiffs deny the allegations contained in paragraph 107 of Public Resource’s 
Counterclaim. 

108. Plaintiffs deny the allegations contained in paragraph 108 of Public Resource’s 
Counterclaim. 

109. Plaintiffs deny the allegations contained in paragraph 109 of Public Resource’s 
Counterclaim. 

110. Plaintiffs deny the allegations contained in paragraph 110 of Public Resource’s 
Counterclaim. 

111. Plaintiffs deny the allegations contained in paragraph 111 of Public Resource’s 
Counterclaim. 

112. Plaintiffs deny the allegations contained in paragraph 112 of Public Resource’s 
Counterclaim. 

113. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 113 of Public Resource’s Counterclaim, and therefore deny same leaving 

-18- 


JA2203 


Case l:14-cv-00857-CRC Document 14 Filed 08/21/14 Page 19 of 25 
USCA Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 445 of 573 

Defendant to its proofs. 

114. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 114 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

115. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 115 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

COUNT I 

[Declaratory Relief Pursuant to 28 U.S.C. § 2201, et seq. (Declaratory Judgment Act) and 
Title 17 U.S.C. (Copyright Act of 1976)] 

116. Plaintiffs repeat and reallege paragraphs 1 through 115 above. 

117. The allegations contained in paragraph 117 of Public Resource’s Counterclaim 
are legal conclusions to which no response is required. 

118. The allegations contained in paragraph 118 of Public Resource’s Counterclaim 
are legal conclusions to which no response is required. By way of a further answer, no action by 
Public Resource is necessary to provide the public with the ability to read the 1999 Standards. 

119. The allegations contained in paragraph 119 of Public Resource’s Counterclaim 
are legal conclusions to which no response is required. 

120. The allegations contained in paragraph 120 of Public Resource’s Counterclaim 
are legal conclusions to which no response is required. 

121. The allegations contained in paragraph 121 of Public Resource’s Counterclaim 
are legal conclusions to which no response is required. 

122. The allegations contained in paragraph 122 of Public Resource’s Counterclaim 
are legal conclusions to which no response is required. By way of a further answer, Plaintiffs 
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state that OMB Circular A-119 and the October 2, 2013 report of the Office of the Federal 
Register, National Archives and Records make clear that copyrighted documents do not lose 
their status and protectable intellectual property through the government action of incorporation 
by reference. 

123. The allegations contained in paragraph 123 of Public Resource’s Counterclaim 
are legal conclusions to which no response is required. By way of a further answer, Plaintiffs 
state that they own the copyright in the 1999 Standards. Plaintiffs do not lose their copyright 
protection in the 1999 Standards as a result of incorporation by reference by the government. 

124. The allegations contained in paragraph 124 of Public Resource’s Counterclaim 
are legal conclusions to which no response is required. 

125. The allegations contained in paragraph 125 of Public Resource’s Counterclaim 
are legal conclusions to which no response is required. 

126. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 126 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

127. The allegations contained in the first sentence of paragraph 127 of Public 
Resource’s Counterclaim are legal conclusions to which no response is required. Plaintiffs admit 
the allegations contained in the second sentence of paragraph 127 of Public Resource’s 
Counterclaim. The allegations contained in the third sentence of paragraph 127 of Public 
Resource’s Counterclaim are legal conclusions to which no response is required. Plaintiffs deny 
the allegations contained in the fourth sentence of paragraph 127 of Public Resource’s 
Counterclaim. 

128. Plaintiffs deny the allegations contained in paragraph 128 of Public Resource’s 
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Counterclaim. 

129. Plaintiffs deny the allegations contained in paragraph 129 of Public Resource’s 
Counterclaim. 

130. Plaintiffs admit the allegations contained in paragraph 130 of Public Resource’s 
Counterclaim, except deny that Public Resource posted Plaintiffs’ 1999 Standard to Defendant’s 
website in a searchable format. 

131. Plaintiffs deny the allegations contained in paragraph 131 of Public Resource’s 
Counterclaim. 

132. Plaintiffs lack information sufficient to form a belief as to the allegations 
contained in paragraph 132 of Public Resource’s Counterclaim, and therefore deny same leaving 
Defendant to its proofs. 

133. Plaintiffs do not understand what Public Resource means by the term “incorporate 
into,” and therefore deny the allegations contained in paragraph 133 of Public Resource’s 
Counterclaim, leaving Defendant to its proofs. By way of a further answer, Plaintiffs cannot 
purport to answer Public Resource’s allegations on behalf of what some government(s) or 
government entity(ies) may or may not do in the future with other versions of the Standards for 
Educational and Psychological Testing. 

134. The allegations contained in the paragraph 134 of Public Resource’s 

Counterclaim are legal conclusions to which no response is required. By way of a further 
answer, Plaintiffs do not understand what Public Resource means by the term “incorporated 
standards,” and therefore deny the allegations contained in paragraph 134 of Public Resource’s 
Counterclaim, leaving Defendant to its proofs. 

135. The allegations contained in the paragraph 135 of Public Resource’s 
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Counterclaim are legal conclusions to which no response is required. 

136. Plaintiffs admit the allegations contained in paragraph 136 of Public Resource’s 
Counterclaim. 

137. Plaintiffs deny the allegations contained in paragraph 137 of Public Resource’s 
Counterclaim. 

Affirmative Defenses to Counterclaim 

First Affirmative Defense 

138. Public Resource’s counterclaim is barred in whole or in part for failure to state a 
claim for which relief may be granted. 

Second Affirmative Defense 

139. Public Resource’s counterclaim is unnecessary in light of Defendant’s denial of 
liability for Plaintiffs’ claim. 

Third Affirmative Defense 

140. Public Resource’s counterclaim is redundant of Defendant’s affirmative defenses 
to Plaintiffs’ claim. 

Fourth Affirmative Defense 

141. The “unclean hands doctrine” derives from the equitable maxim that “he who 
comes into equity must come with clean hands.” 

142. The unclean hands doctrine “closes the doors of a court of equity to one tainted 
with inequitableness or bad faith relative to the matter in which he seeks relief.” 

143. The unclean hands doctrine requires that “one seeking relief must have acted 
fairly and without fraud or deceit as to the controversy at issue.” 

144. “Any willful act concerning the cause of action which rightfully can be said to 
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transgress equitable standards of conduct is sufficient cause for invocation of the [unclean hands] 
maxim ...” 

145. Public Resource willfully published the 1999 Standards to Public Resource’s 
https://law.resource.org website, without legal justification or permission from Plaintiffs to do 
so. 

146. Public Resource’s willful, bad faith infringement and contributory infringement of 
Plaintiffs’ work, the 1999 Standards, transgressed equitable standards of conduct. Public 
Resource further acted unfairly and with fraud and deceit as to the controversy presently before 
the Court. 

147. Public Resource’s counterclaim for declaratory relief is therefore barred by the 
doctrine of unclean hands. 

Fifth Affirmative Defense 

148. Public Resource’s counterclaim is inapplicable, in view of Defendant’s admission 
of infringement (see ( J[ 127 of Defendant’s counterclaim). 

Sixth Affirmative Defense 

149. The relief requested in Public Resource’s counterclaim is contrary to the public 
policy of affording adequate and appropriate protection to works that are subject to copyright. 

// 

// 

// 

// 

// 

// 
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Seventh Affirmative Defense 


150. Public Resource’s counterclaim is not the proper subject of a trial before a jury. 

Respectfully submitted, 

OBLON, SPIVAK, MCCLELLAND, 

MAIER & NEUSTADT,LLP 


/s/ Jonathan Hudis _ 

Dated: August 21,2014 By: Jonathan Hudis (DC Bar #418872) 

Kathleen Cooney-Porter (DC Bar # 434526) 
OBLON, SPIVAK, MCCLELLAND, 
MAIER & NEUSTADT, LLP 
1940 Duke Street 
Alexandria, VA 22314 
Tel. (703) 413-3000 
Fax (703) 413-2220 
E-Mailjhudis@oblon.com 
E-Mail kcooney-porter@oblon.com 


Attorneys for Plaintiffs 

AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION, INC. 

AMERICAN PSYCHOLOGICAL 
ASSOCIATION, INC. 

NATIONAL COUNCIL ON 
MEASUREMENT IN EDUCATION, INC. 
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PRELIMINARY STATEMENT 

Plaintiffs, the American Educational Research Association, Inc. (“AERA”), the American 
Psychological Association, Inc. (“APA”), and the National Council on Measurement in 
Education, Inc. (“NCME”) (collectively, “Plaintiffs” or the “Sponsoring Organizations”), submit 
this Memorandum of Points and Authorities in support of Plaintiffs’ Motion for Summary 
Judgment and the entry of a Permanent Injunction. 

The Sponsoring Organizations’ work, the “Standards for Educational and Psychological 
Testing (1999 ed.)” (the “1999 Standards”) were written at great expense, including many years 
of volunteer effort given by a select group of the leading minds in educational and psychological 
testing of their time. Until recently updated by the 2014 edition, for fifteen years the 1999 
Standards were considered the foremost authority of best practices in the preparation and 
administration of tests; particularly in the areas of validity, reliability, and fairness. The 1999 
Standards continue to have enduring value for those in the testing profession who (i) need to 
know the state of best testing practices as they existed between 1999 and 2014, (ii) believe they 
still may be held accountable to the guidance of the 1999 Standards even now, and/or (iii) study 
the changes in best testing practices over time. 

Defendant, Public.Resource.Org, Inc. (“Public Resource”), by assertion of a “species of 
mutant copyright law,” 1 has made itself judge, jury, and executioner of the Sponsoring 
Organizations’ rights in the 1999 Standards - merely due to the happenstance that the Standards 
were cited in federal and state regulations. Without the Sponsoring Organizations’ permission, 
Public Resource created a PDF (Acrobat Reader) version of the 1999 Standards and posted the 
PDF file to two unrestricted Internet websites. Moreover, Public Resource posted the 1999 

1 Dastar Corp. v. Twentieth Century Fox Film Corp., 539 U.S. 23, 34 (2003) (castigating the plaintiffs for asserting 
a claim of reverse passing off under Trademark Act Section 43(a), 15 U.S.C. § 1125(a) that “would create a species 
of mutant copyright law that limits the pubhc’s ‘federal right to ‘copy and to use’ expired copyrights”.). 
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per copy. From 2000 to 2014, except for the near two-year period during which Public Resource 
posted unauthorized copies online and sales diminished significantly, income generated from 
sales of the 1999 Standards, on average, had been approximately in excess of $127,000 per year 
(Levine Decl., 17-18 Exh. OOO). 

After the 2014 Standards were published in the late summer of 2014, AERA for a time 
discontinued sales of the 1999 Standards. This was to encourage sales of the newly-revised 
edition - the 2014 Standards (Levine Decl., K 19, Exh. PPP). However, so long as purchasers are 
made aware that it is no longer the current edition, the 1999 Standards do have an enduring value 
for those in the testing and assessment profession who (i) need to know the state of best testing 
practices as they existed between 1999 and 2014, (ii) believe they still may be held accountable 
to the guidance of the 1999 Standards even now, and/or (iii) study the changes in best testing and 
assessment practices over time. For these reasons, in the summer of 2015 AERA resumed sales 
of the 1999 Standards (Levine Deck, f 20, Exh. QQQ). 

The Sponsoring Organizations do not keep any of the proceeds generated from the sales 
of the Standards. Rather, the income from these sales is used by the Sponsoring Organizations to 
offset their development and production costs and to generate funds for subsequent revisions. 
This allows the Sponsoring Organizations to develop up-to-date, high quality Standards that 
otherwise would not be developed due to the time and effort that goes into producing them 
(Levine Decl., | 21; Geisinger Decl., | 22; Camara Deck, f 19; Ernesto Deck, | 31). Without 
receiving at least some moderate income from the sales of the Standards to offset their 
production costs and to allow for further revisions, it is very likely that the Sponsoring 
Organizations would no longer undertake to periodically update them, and it is u nk nown who 
else would (Levine Deck, | 22; Ernesto Deck, | 32; Wise Deck, 124; Geisinger Deck, U 22). 
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At one time, funding for the Standards revision process from third party sources {e.g., 
governmental agencies, foundations, other associations interested in testing and assessment 
issues, etc.) was considered. However, this option was not seriously considered as the difficulty 
and/or potential conflicts of interest in doing so left the Sponsoring Organizations to conclude 
that financial support for the Standards revisions should be self-funding - that is, from the sale of 
prior editions of the Standards (Levine Decl., ][ 23; Camara Decl., ][ 20). 

Due to the small membership size of Plaintiff NCME, and the relative minor portion of 
the membership of Plaintiffs AERA and APA who devote their careers to testing and assessment, 
it is highly unlikely that the members of the Sponsoring Organizations will vote for a dues 
increase to fund future Standards revision efforts if Public Resource successfully defends this 
case and is allowed to post the Standards online for the public to download or print for free. As a 
result, the Sponsoring Organizations would likely abandon their practice of periodically updating 
the Standards (Levine Decl., ]j 24; Camara Decl., 24; Geisinger Decl., ]j 23; Ernesto Decl., ][ 
33). 

Plaintiffs’ Ownership of the Copyright in the Standards 

The Plaintiffs are joint owners of the copyright in and to the Standards. The Standards 
were registered with the U.S. Register of Copyrights under Registration Number TX 5-100-196, 
having an effective date of December 8, 1999 (Levine Decl., H 25, Exh. RRR). A supplementary 
copyright registration for the Standards was issued by the U.S. Register of Copyrights under 
Supplementary Registration Number TX 6-434-609, having an effective date of February 25, 
2014 (Levine Decl., 126, Exh. SSS). 

The Joint Committee that authored the 1999 Standards comprised 16 members (Levine 
Decl., Uf 27-28, Exh. TTT). Except for Manfred Meier (who could not be located, nor could his 
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UNITED STATES DISTRICT COURT 
FOR THE DISTRICT OF COLUMBIA 


AMERICAN EDUCATIONAL RESEARCH ) 

ASSOCIATION, INC., AMERICAN ) 

PSYCHOLOGICAL ASSOCIATION, INC., ) 

and NATIONAL COUNCIL ON ) 

MEASUREMENT IN EDUCATION, INC., ) Civil Action No. 1:14-cv-00857-TSC-DAR 

) 

Plaintiffs, ) PLAINTIFFS’ STATEMENT OF 

) MATERIAL FACTS IN SUPPORT 
v. ) OF THEIR MOTION FOR 

) SUMMARY JUDGMENT 

PUBLIC.RESOURCE.ORG, INC., ) 

) 

Defendant. ) 

_ ) 

Pursuant to the Local Rule 7(h), Plaintiffs, American Educational Research Association, 
Inc. (“AERA”), American Psychological Association, Inc. (“APA”) and National Council on 
Measurement in Education, Inc. (“NCME”) (collectively, “Plaintiffs” or the “Sponsoring 
Organizations”) submit, in support of their Motion for Summary Judgment, a Statement of 
Material Facts as to which there are no genuine issues that otherwise would be left for trial: 

I. Plaintiffs 

1. Plaintiffs, AERA, APA, and NCME, are District of Columbia not-for-profit 
corporations (Levine Decl., ]f 4; Ernesto Decl., ]f 3; Wise Decl., K 3). 

2. AERA is the major national scientific society for research on education and 
learning. AERA’s mission is to advance knowledge about education, to encourage scholarly 
inquiry related to education, and to promote the use of research to improve education and serve 
the public good (Levine Decl., 15). 

3. APA is the largest scientific and professional organization representing 
psychology in the United States. APA is the world’s largest association of psychologists and 
counts a vast number of researchers, educators, clinicians, consultants and students among its 
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generated from sales of the 1999 Standards, on average, had been approximately in excess of 
$127,000 per year (Levine Decl., flf 17-18 Exh. OOO). 

35. After the 2014 Standards were published in the late summer of 2014, AERA for a 
time discontinued sales of the 1999 Standards. This was to encourage sales of the newly-revised 
edition - the 2014 Standards (Levine Decl., K 19, Exh. PPP). However, so long as purchasers are 
made aware that it is no longer the current edition, the 1999 Standards do have an enduring value 
for those in the testing and assessment profession who (i) need to know the state of best testing 
practices as they existed between 1999 and 2014, (ii) believe they still may be held accountable 
to the guidance of the 1999 Standards even now, and/or (iii) study the changes in best testing and 
assessment practices over time. For these reasons, in the summer of 2015 AERA resumed sales 
of the 1999 Standards (Levine Deck, f 20, Exh. QQQ). 

36. The Sponsoring Organizations do not keep any of the proceeds generated from 
the sales of the Standards. Rather, the income from these sales is used by the Sponsoring 
Organizations to offset their development and production costs and to generate funds for 
subsequent revisions. This allows the Sponsoring Organizations to develop up-to-date, high 
quality Standards that otherwise would not be developed due to the time and effort that goes into 
producing them (Levine Deck, If 21; Geisinger Deck, ]f 22; Camara Deck, f 19; Ernesto Deck, ]f 
31). 

37. Without receiving at least some moderate income from the sales of the Standards 
to offset their production costs and to allow for further revisions, it is very likely that the 
Sponsoring Organizations would no longer undertake to periodically update them, and it is 
u nk nown who else would (Levine Deck, ]f 22; Ernesto Deck, ][ 32; Wise Deck, ][ 24; Geisinger 
Deck, 1 22). 
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IN THE UNITED STATES DISTRICT COURT 
FOR THE DISTRICT OF COLUMBIA 


AMERICAN EDUCATIONAL RESEARCH ) 

ASSOCIATION, INC., AMERICAN ) 

PSYCHOLOGICAL ASSOCIATION, INC., ) 

and NATIONAL COUNCIL ON ) 

MEASUREMENT IN EDUCATION, INC., ) 

) 

Plaintiffs/Counterdefendants, ) 

) 

v. ) 

) 

PUBLIC.RESOURCE.ORG, INC., ) 

) 

Defendant/Counterclaimant. ) 

_ ) 


Civil Action No. l:14-cv-00857-TSC-DAR 


DEFENDANT-COUNTERCLAIMANT 
PUBLIC.RESOURCE.ORG, INC.’S 
AMENDED RESPONSES TO 
PLAINTIFFS- 

COUNTERDEFENDANTS’ FIRST SET 
OF INTERROGATORIES (NOS 1-8) 


Pursuant to Rules 26 and 33 of the Federal Rules of Civil Procedure and the Local Civil 


Rules of Civil Procedure of this Court, Defendant Public.Resource.Org, Inc. (“Public Resource”) 
hereby responds to Plaintiffs’ American Educational Research Association, Inc., American 
Psychological Association, Inc., and National Council on Measurement in Education, Inc., First 
Set of Interrogatories. 


GENERAL OBJECTIONS 


1. Public Resource objects to the interrogatories to the extent that they are overly 
broad, unduly burdensome, or oppressive, or to the extent they are inconsistent with, or purport 
to impose obligations on Public Resource beyond those set forth by the Federal Rules of Civil 
Procedure, the Local Rules of the United States District Court for the District of Columbia, the 
Federal Rules of Evidence, or any applicable regulations and case law, particularly to the extent 
that compliance would force Public Resource to incur a substantial expense that outweighs any 
likely benefit of the discovery. Public Resource’s responses, regardless of whether they include 
a specific objection, do not constitute an adoption or acceptance of the definitions and 
instructions that Plaintiffs seek to impose. 


I 
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2. Public Resource objects to the interrogatories to the extent that they seek 
documents and information that are neither relevant to the Action nor reasonably calculated to 
lead to the discovery of admissible evidence. Public Resource objects to the interrogatories to 
the extent that they seek documents and information that are not in Public Resource’s possession, 
custody or control. Public Resource objects to the interrogatories on the ground that they seek to 
impose obligations on Public Resource that are unduly burdensome, especially to the extent that 
requested information is publicly available or burdensome to search for or obtain. Public 
Resource further objects to the extent that the interrogatories are overbroad and that their number 
exceeds the number that the Federal Rules of Civil Procedure authorize. 

3. Public Resource objects to the interrogatories to the extent that they seek 
information that falls under the attorney-client privilege, work-product doctrine, common interest 
privilege, or other applicable privileges or protections. Public Resource will not provide such 
information, and any inadvertent production is not a waiver of any applicable privilege or 
protection. 

4. Public Resource objects to the interrogatories, and each and every instruction and 
definition, to the extent that Plaintiffs seek information that is not limited to a relevant and 
reasonable period of time. 

5. Public Resource objects to the interrogatories to the extent they are 
argumentative. 

6. Public Resource objects to the interrogatories to the extent they are cumulative 
and/or duplicative of any other of Plaintiffs’ discovery requests. 

7. Public Resource objects to the definition of “Public Resource” on the grounds that 
it is vague, ambiguous, overbroad, and unduly burdensome, particularly to the extent that it 
purports to include any affiliates or other persons when such persons are acting outside of a 
capacity of representing Public Resource. 

8. Public Resource objects to the term “Public Resource Website” on the grounds 
that it is vague, ambiguous, overbroad, and unduly burdensome, particularly to the extent it fails 
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to identify which websites Plaintiffs consider “owned, controlled or operated” by Public 
Resource, other than law.resource.org, public.resource.org, house.resource.org, and 
bulk.resource.org. 

9. Public Resource objects to the definition of “communication” to the extent it 
includes information protected by the attorney-client privilege, the attorney work product 
doctrine, or any other applicable privilege or protection as provided by law. Public Resource 
further objects to the definitions of this term as seeking to impose obligations broader than, or 
inconsistent with, the Federal Rules, the Local Rules, and/or the law of this Circuit. Public 
Resource will respond to the interrogatories using the ordinary meaning of “communications” 
and the scope of this term given by Fed. R. Civ. P. 33 and 34. 

10. Public Resource objects to the definition of “identify” to the extent it seeks to 
impose obligations broader than, or inconsistent with, the Federal Rules, the Local Rules, and/or 
the law of this Circuit. Public Resource will respond to the interrogatories using the ordinary 
meaning of “identify” and the scope of this term given by Fed. R. Civ. P. 33. 

11. Public Resource objects to the extent the requests are premature. It reserves the 
right to amend or supplement its responses as the Action proceeds. 

12. Public Resource’s responses to these interrogatories are made without waiving, or 
intending to waive, but on the contrary, preserving and intending to preserve: (a) the right to 
object, on the grounds of competency, privilege, relevance or materiality, or any other proper 
grounds, to the use of any documents or other information for any purpose in whole or in part, in 
any subsequent proceeding in this action or in any other action; (b) the right to object on any and 
all grounds, at any time, to other requests for production, interrogatories, or other discovery 
procedures involving or relating to the subject matter of the interrogatory to which Defendants 
have responded here; and (c) the right at any time to revise, correct, add to, or clarify any of the 
responses made here. 
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RESPONSES TO INTERROGATORIES 


INTERROGATORY NO. 1 ; 

Identify and describe how Public Resource obtained any printed version or versions of 
the 1999 Standards. 

RESPONSE TO INTERROGATORY NO. 1 : 

Public Resource incorporates its general objections as if fully set forth here. Public 
Resource objects to this interrogatory to the extent it purports to impose upon Public Resource 
obligations broader than, or inconsistent with, the Federal Rules of Civil Procedure, local rules, 
Court Orders for this proceeding, or any applicable regulations and case law. 

Subject to and without waiving the foregoing objections. Public Resource responds as 
follows: 

Public Resource purchased a printed copy from “thebookgrove,” a used book seller, on 
May 17, 2012. Pursuant to Federal Rule of Civil Procedure 33(d), Public Resource will produce 
the invoice or invoices for such orders that are in its custody, possession, or control. Public 
Resource believes that to the extent details such as dates, payment amounts, and product names 
are available, they may be derived from this invoice or invoices. 

INTERROGATORY NO. 2 : 

Identify when the 1999 Standards were first and last posted or published to a Public 
Resource Website, and identify the particular Public Resource Website(s) to which the 1999 
Standards were posted or published. 

RESPONSE TO INTERROGATORY NO. 2 : 

Public Resource incorporates its general objections as if fully set forth here. Public 
Resource objects to this interrogatory to the extent it purports to impose upon Public Resource 
obligations broader than, or inconsistent with, the Federal Rules of Civil Procedure, local rules, 
Court Orders for this proceeding, or any applicable regulations and case law. Public Resource 
objects to this request to the extent that it seeks information that is equally available to Plaintiffs 
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from other sources that are more convenient, less burdensome and/or less expensive, particularly 
to the extent the information sought is publicly available. 

Subject to and without waiving the foregoing objections, Public Resource responds as 
follows: 

The 1999 Standard was first posted to the law.resource.org website on July 11, 2012. On 
that date, the 1999 Standard was also posted by Public Resource on the Public Safety collection 
on the Internet Archive. 

The 1999 Standard was last posted to a Public Resource Website on June 10, 2014. The 
1999 Standard was also removed from public view on the Internet Archive on that date. 

INTERROGATORY NO. 3 : 

Identify and describe the process Public Resource used to digitize or convert to digital 
format the 1999 Standards from paper format, including any quality control measures Public 
Resource used to prevent the content of the 1999 Standards from being altered. 

RESPONSE TO INTERROGATORY NO. 3 : 

Public Resource incorporates its general objections as if fully set forth here. Public 
Resource objects to this interrogatory to the extent it purports to impose upon Public Resource 
obligations broader than, or inconsistent with, the Federal Rules of Civil Procedure, local rules, 
Court Orders for this proceeding, or any applicable regulations and case law. Public Resource 
objects to this interrogatory and to the term “digitize” as vague and ambiguous. 

Subject to and without waiving the foregoing objections, Public Resource responds as 
follows: 

Public Resource procures standards, including the 1999 Standard, as paper documents. It 
then disassembles them, removing any spines, stuffing, staples, or other extraneous materials. If 
necessary, it trims the documents to give them an even border. It then scans the documents on a 
Xerox 4250 scanner at 300 or 400 dots per inch. It names each file in a standard manner, 
including the standard setting organization, the standard number, and the date. For example, one 
name is “aera.standards.l999.pdf.” 
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Public Resource appends a cover sheet to each file consisting of the name of the standard 
setting organization, the date, the title, and (most importantly) the specific section of law that 
incorporates this specific standard into law. This is an important part of the quality check 
process. If Public Resource is unable to find a specific incorporation, it does nothing further 
with the standard and does not post it. 

The scanning process produces files in PDF format. The files are post-processed to 
optimize the scans and to generate Optical Character Recognition (OCR) on the text. Public 
Resource then double-checks the IBR reference(s), puts a cover sheet on the files, and stamps 
metadata into the headers. Public Resource then posts these files on its website as well as the 
Internet Archive using the HTTPS (secure HTTP) protocol. Public Resource also makes the files 
available using FTP and Rsync. 

INTERROGATORY NO. 4 : 

Identify all persons who and companies that participated on Public Resource’s behalf 
(including Public Resource itself) in digitizing or converting a paper version of the 1999 
Standards to digital format, and describe the nature of each person’s and/or company’s 
participation. 

RESPONSE TO INTERROGATORY NO, 4 : 

Public Resource incorporates its general objections as if fully set forth here. Public 
Resource objects to this interrogatory to the extent it purports to impose upon Public Resource 
obligations broader than, or inconsistent with, the Federal Rules of Civil Procedure, local rules, 
Court Orders for this proceeding, or any applicable regulations and case law. Public Resource 
objects to this interrogatory as outside the scope of discovery to the extent it calls for information 
that is not within Public Resource’s knowledge. Public Resource objects to this interrogatory as 
overly broad and unduly burdensome to the extent it seeks information about entities other than 
Public Resource. Public Resource objects to this interrogatory to the extent it calls for the 
disclosure of information that is protected by any individual’s right of privacy. 
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Subject to and without waiving the foregoing objections. Public Resource responds as 
follows: 

Public Resource incorporates by reference its response to Interrogatory No. 3. 

Contact information for each entity is as follows: 

Carl Malamud 
Publ ic. Resource. Org 
1005 Gravenstein Hwy N 
Sebastopol, CA 95472 

INTERROGATORY NO. 5 : 

Identify and describe, by month and year starting from the date that the 1999 Standards 
were first posted on or published to a Public Resource Website or Public Resource Websites, the 
number of visitors who viewed and/or accessed the 1999 Standards on that website or those 
websites. 

RESPONSE TO INTERROGATORY NO, 5 : 

Public Resource incorporates its general objections as if fully set forth here. Public 
Resource objects to this interrogatory to the extent it purports to impose upon Public Resource 
obligations broader than, or inconsistent with, the Federal Rules of Civil Procedure, local rules, 
Court Orders for this proceeding, or any applicable regulations and case law. Public Resource 
objects to this interrogatory and to the term “viewed and/or accessed” as vague and ambiguous. 
Public Resource objects to this interrogatory as seeking information not relevant to any party’s 
claims or defenses and not reasonably calculated to lead to the discovery of admissible evidence 
to the extent that the term “accessed” means “viewed.” Public Resource objects to this 
interrogatory to the extent that the scope of the information sought is not limited to a relevant 
and reasonable period of time. 
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Subject to and without waiving the foregoing objections, to the extent the information 
sought is available, Public Resource will produce and identify non-privileged documents that 
exist within its possession, custody, and control from which the response to this interrogatory 
may be derived. 

AMENDED RESPONSE TO INTERROGATORY NO. 6 : 

Public Resource incorporates its general objections as if fully set forth here. Public 
Resource objects to this interrogatory to the extent it purports to impose upon Public Resource 
obligations broader than, or inconsistent with, the Federal Rules of Civil Procedure, local rules, 
Court Orders for this proceeding, or any applicable regulations and case law. Public Resource 
objects to this interrogatory and to the term “viewed and/or accessed” as vague and ambiguous. 
Public Resource objects to this interrogatory as seeking information not relevant to any party’s 
claims or defenses and not reasonably calculated to lead to the discovery of admissible evidence 
to the extent that the term “accessed” means “viewed.” Public Resource objects to this 
interrogatory to the extent that the scope of the information sought is not limited to a relevant 
and reasonable period of time. 

After conference, the parties agree as follows: 

“Accessed” means to digitally retrieve or open an electronic file or data. 

“View(ed)” means the act of seeing or examining. 

“Downloaded” means a user reproducing an electronic file by saving a reproduction of the 
file to a location on the user’s device with the intent to facilitate permanent ready access 
until the user deletes the file. This definition of “download” includes use of functions such 
as “Save” and “Save As,” but does not include printing physical hardcopies, taking 
screenshots, or cache reproductions such as “Temporary Internet files.” 

Based on the parties’ agreement on the definitions of “viewed” and “accessed,” Public 
Resource responds as follows: 

Public Resource recorded on the Public.Resource.org website the following number of 
HyperText Transfer Protocol (HTTP) requests for the filename “aera.standards. 1999.pdf’ for 
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each month and date below. In calculating the number of HTTP requests. Public Resource 
counted each successful full retrieval request (“status code 200”) as one request and all partial 
retrieval requests (“status code 206”) within the same hour as one request (under the assumption 
that each set was one device making a series of partial retrieval requests that added up to one full 
retrieval). 


2013-08 

18 

2013-09 

58 

2013-10 

259 

2013-11 

260 

2013-12 

331 

2014-01 

564 

2014-02 

471 

2014-03 

536 

2014-04 

633 

2014-05 

741 

2014-06 

293 

2014-07 

69 

2014-08 

48 

2014-09 

30 

2014-10 

50 


Public Resource recorded on the Public.Resource.org website the following number of 

File Transfer Protocol (FTP) requests for the filename “aera.standards.l999.pdf’ for each month 

and date below. 

2013-06: 1 

2013-07: 2 

2013-08: 1 

2013-09: 3 

2013-10: 3 

2013-11: 4 

2013- 12: 8 

2014- 03: 6 

2014-04: 5 

2014-05: 4 

2014-06: 1 

2014-08: 1 

2014-09: 2 

2014-10: 1 
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Public Resource recorded on the Public.Resource.org website the following number of 

Rsync (remote sync) protocol requests for the filename “aera.standards.l999.pdf’ for each month 

and date below. 

2013-04: 1 

2013-11: 1 

On June 10, 2014, at Plaintiffs’ request, Public Resource replaced on its website the 
document which had the filename “aera.standards. 1999.pdf’ with a stub document explaining 
this litigation. Because the stub document has the same filename, retrievals of that document 
also appear in this set of records. 

INTERROGATORY NO. 6 : 

Identify the number of times the 1999 Standards were downloaded from a Public 
Resource Website or Public Resources Websites, and identify the particular Public Resource 
Website(s) from which the 1999 Standards were downloaded. 

RESPONSE TO INTERROGATORY NO. 7 : 

Public Resource incorporates its general objections as if fully set forth here. Public 
Resource objects to this interrogatory to the extent it purports to impose upon Public Resource 
obligations broader than, or inconsistent with, the Federal Rules of Civil Procedure, local rules, 
Court Orders for this proceeding, or any applicable regulations and case law. Public Resource 
objects to this interrogatory and to the term “downloaded” as vague and ambiguous. Public 
Resource objects to this interrogatory as seeking information not relevant to any party’s claims 
or defenses and not reasonably calculated to lead to the discovery of admissible evidence to the 
extent that the term “downloaded” means “viewed.” Public Resource objects to this 
interrogatory to the extent that the scope of the information sought is not limited to a relevant 
and reasonable period of time. 

Subject to and without waiving the foregoing objections, to the extent the information 
sought is available. Public Resource will produce and identify non-privileged documents that 
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exist within its possession, custody, and control from which the response to this interrogatory 
may be derived. 

AMENDED RESPONSE TO INTERROGATORY NO. 8 : 

Public Resource incorporates its general objections as if fully set forth here. Public 
Resource objects to this interrogatory to the extent it purports to impose upon Public Resource 
obligations broader than, or inconsistent with, the Federal Rules of Civil Procedure, local rules, 
Court Orders for this proceeding, or any applicable regulations and case law. Public Resource 
objects to this interrogatory and to the term “downloaded” as vague and ambiguous. Public 
Resource objects to this interrogatory as seeking information not relevant to any party’s claims 
or defenses and not reasonably calculated to lead to the discovery of admissible evidence to the 
extent that the term “downloaded” means “viewed.” Public Resource objects to this 
interrogatory to the extent that the scope of the information sought is not limited to a relevant 
and reasonable period of time. 

After conference, the parties agree as follows: 

“Accessed” means to digitally retrieve or open an electronic file or data. 

“View(ed)” means the act of seeing or examining. 

“Downloaded” means a user reproducing an electronic file by saving a reproduction of the 
file to a location on the user’s device with the intent to facilitate permanent ready access 
until the user deletes the file. This definition of “download” includes use of functions such 
as “Save” and “Save As,” but does not include printing physical hardcopies, taking 
screenshots, or cache reproductions such as “Temporary Internet files.” 

Based on the parties’ agreement on the definitions of “accessed,” “viewed” and 
“downloaded,” Public Resource responds as follows: 

Subject to and without waiving the foregoing objections, to the best of its knowledge at 
this time. Public Resource has no information responsive to this request specific to the act of 
downloading (as opposed to the act of accessing, which Public Resource addresses in its 
response to Interrogatory No. 5). Public Resource’s investigation is ongoing, and to the extent it 
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locates any non-privileged documents from which responsive information may be derived, it will 
produce them pursuant to Federal Rule of Civil Procedure 33(d). 

INTERROGATORY NO. 7 : 

Identify and describe all instances of which you are aware in which a third party, after 
downloading the 1999 Standards from a Public Resource Website, posted the 1999 Standards 
online to a website other than a Public Resource Website, made further reproductions of the 1999 
Standards, or created derivative works based on the 1999 Standards. 

RESPONSE TO INTERROGATORY NO. 9 : 

Public Resource incorporates its general objections as if fully set forth here. Public 
Resource objects to this interrogatory to the extent it purports to impose upon Public Resource 
obligations broader than, or inconsistent with, the Federal Rules of Civil Procedure, local rules, 
Court Orders for this proceeding, or any applicable regulations and case law. Public Resource 
objects to this interrogatory and to the term “downloading” as vague and ambiguous. Public 
Resource objects to this interrogatory to the extent that the scope of the information sought is not 
limited to a relevant and reasonable period of time. 

Subject to and without waiving the foregoing objections, Public Resource responds that it 
is not aware of any information responsive to this interrogatory. Public Resource’s investigation 
is ongoing, and to the extent it locates any non-privileged documents from which responsive 
information may be derived, it will produce them pursuant to Federal Rule of Civil Procedure 
33(d). 

AMENDED RESPONSE TO INTERROGATORY NO. 10 : 

Public Resource incorporates its general objections as if fully set forth here. Public 
Resource objects to this interrogatory to the extent it purports to impose upon Public Resource 
obligations broader than, or inconsistent with, the Federal Rules of Civil Procedure, local rules. 
Court Orders for this proceeding, or any applicable regulations and case law. Public Resource 
objects to this interrogatory and to the term “downloading” as vague and ambiguous. Public 
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Resource objects to this interrogatory to the extent that the scope of the information sought is not 
limited to a relevant and reasonable period of time. 

After conference, the parties agree as follows: 

- “Accessed” means to digitally retrieve or open an electronic file or data. 

- “View(ed)” means the act of seeing or examining. 

“Downloaded” means a user reproducing an electronic file by saving a reproduction of the 
file to a location on the user’s device with the intent to facilitate permanent ready access 
until the user deletes the file. This definition of “download” includes use of functions such 
as “Save” and “Save As,” but does not include printing physical hardcopies, taking 
screenshots, or cache reproductions such as “Temporary Internet files.” 

Based on the parties’ agreement on the definitions of “accessed,” “viewed” and 
“downloaded,” Public Resource responds as follows: 

Subject to and without waiving the foregoing objections. Public Resource responds that it 
is not aware of any information responsive to this interrogatory. Public Resource’s investigation 
is ongoing, and to the extent it locates any non-privileged documents from which responsive 
information may be derived, it will produce them pursuant to Federal Rule of Civil Procedure 
33(d). 

INTERROGATORY NO, 8 : 

State the factual and legal basis of each Affirmative and Other Defense to Plaintiffs’ 
Complaint, as asserted in Public Resource’s Counterclaim and Answer filed with the Court on 
July 14, 2014. 

RESPONSE TO INTERROGATORY NO. 11 : 

Public Resource incorporates its general objections as if fully set forth here. Public 
Resource objects to this interrogatory to the extent it purports to impose upon Public Resource 
obligations broader than, or inconsistent with, the Federal Rules of Civil Procedure, local rules, 
Court Orders for this proceeding, or any applicable regulations and case law. Public Resource 
objects to this interrogatory to the extent it seeks disclosure of information that falls under the 
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work product doctrine. Public Resource objects to this interrogatory because it is argumentative. 
Public Resource objects to this interrogatory because it seeks information that is publicly 
available, already known, or equally available to Plaintiffs. Public Resource objects to this 
interrogatory as it seeks “factual and legal basis” at an early stage of the litigation. 


Date: December 15, 2014 FENWICK & WEST LLP 


/s/Andrew P. Bridses _ 

Andrew P. Bridges (admitted) 
abridges@fenwick. com 
555 California Street, 12th Floor 
San Francisco, CA 94104 
Telephone: (415) 875-2300 
Facsimile: (415) 281-1350 

David Halperin (D.C. Bar No. 426078) 
davidhalperindc@gmail .com 

1530 P Street NW 
Washington, DC 20005 
Telephone: (202) 905-3434 

Corynne McSherry (pro hac vice) 
corvnne@eff.org 

Mitchell L. Stoltz (D.C. Bar No. 978149) 
mitch@eff.org 

ELECTRONIC FRONTIER FOUNDATION 

815 Eddy Street 

San Francisco, CA 94109 

Telephone: (415) 436-9333 

Facsimile: (415) 436-9993 

Attorneys for Defendant-Counterclaimant 
PUBLIC.RESOURCE.ORG, INC. 
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VERIFICATION 


I, Carl Malamud, declare: 

I am President of Public.Resource.Org, a nonprofit corporation organized and existing 
under the laws of California, which is the defendant in the above-entitled action, and I have been 
authorized to make this verification on its behalf. 

I have read the foregoing DEFENDANT-COUNTERCLAIMANT 
PUBLIC.RESOURCE.ORG, INC.’S AMENDED RESPONSES TO PLAINTIFF- 
COUNTERDEFENDANTS’ FIRST SET OF INTERROGATORIES and know its contents. 

They are true, except as to those matters identified as on information and belief, and, as to those 
matters, I believe them to be true. 

I declare under penalty of perjury that the foregoing is true and correct. 

Executed this 15th day of December, 2014. 



Carl Malamud 
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Copyright © 1999 by the American Educational 
Research .Association, the American Psychological 
Association, and the National Council on 
Measurement in Education. All rights reserved. 
Except as permitted under the United States 
Copyright Act of 1976, no part of this publication 
may be reproduced or distributed in any form or 
by any means, or stored in a database or retrieval 
system, without the prior written permission of 
the publisher. 

Published by 

American Educational Research Association 
1430 K St, NW, Suite 1200 
Washington, DC 20005 

Library of Congress. Card number: 99066845 
ISBN: 0-935302-25-5 
ISBN-13: 978-0-935302-25-7 

Printed in the United States of America 
First printing in 1999; second, 2002; third, 2004; 
fourth, 2007; fifth, 2008; and sixth, 2011. 

The Standards for Educational and Psychotogfcal 
Testing will be under continuing review by the 
three sponsoring organizations. Comments and 
suggestions will be welcome and should be sent to 
The Committee to Develop Standards for 
Educational and Psychological Testing in care of 
the Executive Office, American Psychological 
Association, 750 First Street, NE, Washington, 

DC 20002-4242. 

Prepared by the 

Joint Committee on Standards for Educational 
and Psychological Testing of the American 
Educational Research Association, the American 
Psychological Association, and the National 
Council on Measurement in Education. 
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There have been five earlier documents from 
three sponsoring organizations guiding the 
development and use of tests. The first of these 
was Technical Recommendations for Psychological 
Tests and Diagnostic Techniques, prepared by 
a committee of the American Psychological 
Association (APA) and published by that 
organization in 1954. The second vvas Technical 
Recommendations for Achievement Tests, prepared 
by a committee representing the American 
Educational Research Association (AERA) 
and the National Council on Measurement 
Used in Education (NCMUE) and published 
by the National Education Association in 
1955. The third, which replaced the earlier 
two, was published by APA in 1966 and 
prepared by a committee representing APA, 
AERA, and the National Council on 
Measurement in Education (NCME) and 
called the Standards for Educational and 
Psychological Tests and Manuals. The fourth, 
Standards for Educational and Psychological 
-Tests, was again a collaboration of AERA, APA 
and NCME, and was published in 1974. The 
fifth. Standards fir Educational and Psychological 
Testing, also a joint collaboration, was pub¬ 
lished in 1985. 

In 1991 APA’s Committee on Psycholo¬ 
gical Tests and Assessment suggested the need 
to revise the 1985 Standards. Representatives 
of AERA, APA and NCME met and discussed 
the revision, principles that should guide 
that revision, and potential Joint Committee 
members. By 1993, the presidents of the 
three organizations appointed members 
and the Committee had its first meeting 
November, 1993. 

The Standards has been developed by a 
joint committee appointed by AERA, APA and 
NCME. Members of the Committee were: 

Eva Baker, co-chair 

Paul Sackett, co-chair 

Lloyd Bond 

Leonard Feldt 


David Goh 
Bert Green 
Edward Haertel 
Jo-Ida Hansen 
Sharon Johnson-Lewis 
Suzanne Lane 
Joseph Matarazzo 
Manfred Meier 
Pamela Moss 
Esteban Olmedo 
Diana Pullin 

From 1993 to 1996 Charles Spielbetger 
served on the Committee as co-chair. Each 
sponsoring organization was permitted 
to assign up to two liaisons to the Joint 
Committee’s project. Liaisons served as the 
conduits between the sponsoring organiza¬ 
tions and the Joint Committee. APAs liaison 
from its Committee on Psychological Tests 
and Assessments changed several times as the 
membership of the Committee changed. 
Liaisons to the Joint Committee: 

AERA -William Mehrens 
APA - Bruce Bracken, Andrew Czopek, 
Rodney Lowman, Thomas Oakland 
NCME - Daniel Eignor 

APA and NCME also had committees 
who served to monitor the process and keep 
relevant parties informed. 

APA Ad Hoc Committee of the Council of 
Representatives: 

Melba Vasquez 
Donald BersofF 
Stephen DeMers 
James Farr 
Bertram Karon 
Nadine Lambert 
Charles Spielberger 

NCME Standards and Test Use Committee: 

Gregory Cizek 
Allen Doolittle 
Le Ann Gamache 
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PREFACE 


Donald Ross Green 
Ellen Julian 
Tracy Mucnz 
Nambury Raju 

A management committee was formed at 
the beginning of this effort. They monitored 
the financial and administrative arrangements 
of the project, and advised the sponsoring 
organizations on such matters. 

Management Committee: 

Frank Farley, APA 
George Madaus, AERA 
Wendy Yen, NCME 
Staffing for the revision included Dianne 
Brown Maranto as project director, and 
Dianne L. Schneider as staff liaison. Wayne J. 
Camara served as project director from 1993 to 
1994. APA’s legal counsel conducted the legal 
review of the Standards. William C. Howell 
and William Mehrens reviewed the standards 
for consistency across chapters. Linda Murphy 
developed the indexing for the book. 

The joint Committee solicited prelimi¬ 
nary reviews of some draft chapters, from rec¬ 
ognized experts. These reviews were primarily 
solicited for the technical and fairness chap¬ 
ters. Reviewers are listed below: 

Marvin Alkin 
Philip Bashook 
Bruce Btoxom 
Jeffery P. Braden 
Robert L. Brennan 
John Callender 
Ronald Cannella 
Lee J. Cronbach 
James Cummins 
John Fremer 
Kurt F. Geisingcr 
Robert M. Guion 
Walter Haney 
Patti L Harrison 
Gerald P. Koocher 
Richard Jeanneret 


Frank Landy 
Ellen Lent 
Robert Linn 
Theresa C. Liu 
Stanford von Mayrhauser 
Milbrey W. McLaughlin 
Samuel Messick 
Craig N. Mills 
Robert J. Mislevy 
Kevin R. Murphy 
Mary Anne Nester 
Maria Pennock-Roman 
Carole Perlman 
Michael Rosenfeld 
Jonathan Sandoval 
Cynthia B. Schmeiser 
Kara Schmitt 
Neal Schmitt 
Richard J. Shavelson 
Lorrie A. Shepard 
Mark E. Swerdlik 
Janet Wall 
Anthony R. Zara 

Draft versions of the Standards were 
widely distributed for public review and 
comment three times during this revision 
effort, providing the Committee with a 
total of nearly 8,000 pages of comments. 
Organizations who submitted comments on 
drafts are listed below. Many individuals 
contributed to the input from each organi¬ 
zation, and although we wish we could 
acknowledge every individual who had input, 
we cannot do so due to incomplete informa¬ 
tion as to who contributed to each organiza¬ 
tions response. The joint Committee could 
not have completed its task without the 
thoughtful reviews of so many professionals. 
Sponsoring Associations 

American Educational Research 
Association (AERA) 

American Psychological Association (APA) 
National Council on Measurement in 
Education (NCME) 
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PREFACE 


Membership Organizations (Scientific, 
Professional, Trade & Advocacy) 

American Association for Higher 
Education (AAHE) 

American Board of Medical Specialties 
(ABMS) 

American Counseling Association (ACA) 
American Evaluation Association (AEA) 
American Occupational Therapy 
Association 

American Psychological Society (APS) 
APA Division of Counseling Psychology 
(Division 17) 

APA Division of Developmental 
Psychology (Division 7) 

APA Division of Evaluation, Measurement, 
and Statistics (Division 5) 

APA Division of Mental Retardation & 
Developmental Disabilities (Division 33) 
APA Division of Pharmacology & 
Substance Abuse (Division 28) 

APA Division of Rehabilitation 
Psychology (Division 22) 

APA Division of School Psychology 
(Division 16) 

Asian American Psychological 
Association (AAPA) 

Association for Assessment in 
Counseling (AAC) 

Association ofTest Publishers (ATP) 
Australian Council for Educational 
Research Limited (ACER) 

Chicago industrial/Organizational 
Psychologists (CIOP) 

Council on Licensure, Enforcement, and 
Regulation (CLEAR), Examination 
Resources & Advisory Committee 
(ERAC) 

Equal Employment Advisory Council 
(EEAC) 

Foundation for Rehabilitation 

Certification, Education and Research 
Human Sciences Research Council, 
South Africa 

International Association for Cross- 
Cultural Psychology (IACCP) 


International Brotherhood of Electrical 
Workers 

International Language Testing Association 
International Personnel Management 
Association Assessment Council 
(IPMAAC) 

Joint Committee on Testing Practices 

ycTP) 

National Association for the Advancement 
of Colored People (NAACP), Legal 
Defense and Educational Fund, Inc. 
National Center for Fair and Open 
Testing (Fairtest) 

National Organization for Competency 
Assurance (NOCA) 

Personnel Testing Council of Metropolitan 
Washington (PTC/MW) 

Personnel Testing Council of Southern 
California (PTC/SC) 

Society for Human Resource Management 
(SHRM) 

Society of Indian Psychologists (SIP) 
Society for Industrial and Organizational 
Psychology (APA Division 14) 

Society for the Psychological Study 
of Ethnic Minority Issues (APA 
Division 45) 

State Collaborative on Assessment & 
Student Standards Technical Guidelines 
for Performance Assessment 
Consortium (TGPA) 
Telecommunications Staffing Forum 
Western Region Intergovernmental 
Personnel Assessment Council 
(WRIPAC) 

Credentialing Boards 

American Board of Physical and Medical 
Rehabilitation 

American Medical Technologists 
Commission on Rehabilitation 
Counselor Certification 
National Board for Certified Counselors 
(NBCC) 

National Board of Examiners in 
Optometry 
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PREFACE 


National Board of Medical Examiners 
National Council of State Boards of 
Nursing 

Government and Federal Agencies 
Army Research Institute (ARI) 

California Highway Patrol, Personnel and 
Training Division, Selection Research 
Program 

City of Dallas, Civil Service Department 
Commonwealth of Virginia, Department 
of Education 

Defense Manpower Data Center 

(DMDC), Personnel Testing Division 
Department of Defense (DOD), Office 
of the Assistant Secretary of Defense 
Department of Education, Office of 
Educational Improvement, National 
Center for Education Statistics 
Department of Justice, Immigration and 
Naturalization Service (INS) 
Department of Labor, Employment and 
Training Administration (DOL/ETA) 
U.S. Equal Employment Opportunity 
Commission (EEOC) 

U.S. Office of Personnel Management 
(OPM), Personnel Resources & 
Development Center 
Test Publishers/Developers 

American College Testing (ACT) 
CTB/McGraw-Hill 
The College Board 
Educational Testing Service (ETS) 
Highland Publishing Company 
Institute for Personality & Ability 
Testing (IPAT) 

Professional Examination Service (PES) 
Academic Institutions 

Center for Creative Leadership 
Gallaudet University, National Task 
Force on Equity in Testing Deaf 
Professionals 

University of Haifa, Israeli Group 
Kansas State University 
National Center on Educational 
Outcomes (NCEO) 


Pennsylvania State University 
University of North Carolina - Charlotte 
University of Southern Mississippi, 
Department of Psychology 
When the Joint Committee completed 
its task of revising the Standards, it then 
submitted its work to the three sponsoring 
organizations for approval. Each organization 
had its own governing body and mechanism 
for approval, as well as definitions for what 
their approval means. 

AERA: This endorsement carries with it 
the understanding that, in general, we 
believe the Standards to represent the 
current consensus among recognized 
professionals regarding expected meas¬ 
urement pracrice. Developers, sponsors, 
publishers, and users of tests should 
observe these Standards. 

APA: The APA’s approval of the 
Standards means the Council adopts 
the document as APA policy. 

NCME: NCME endorses the Standards 
for Educational and Psychological Testing 
and recognizes that the intent of these 
Standards is to promote sound and 
responsible measurement practice. This 
endorsement carries with it a profes¬ 
sional imperative for NCME members 
to attend to the Standards. 

Although the Standards are prescriptive, the 
Standards itself does not contain enforcement 
mechanisms. These standards were formulated 
with the intent of being consistent with other 
standards, guidelines and codes of conduct 
published by the three sponsoring organizations, 
and listed below. The reader is encouraged to 
obtain these documents, some of which have 
references to testing and assessment in specific 
applications or settings. 

The Joint Committee on the 
Standards for Educational and 
Psychological Testing 
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INTRODUCTION 


Educational and psychological testing and 
assessment are among the most important 
contributions of behavioral science to our 
society, providing fundamental and signifi¬ 
cant improvements over previous practices. 
Although not all tests are well-developed nor 
are all testing practices wise and beneficial, 
there is extensive evidence documenting the 
effectiveness of well-constructed tests for uses 
supported by validity evidence. The proper 
use of tests can result in wiser decisions about 
individuals and programs than would be the 
case without their use and also can provide a 
route to broader and more equitable access to 
education and employment. The improper 
use of tests, however, can cause considerable 
harm co test takers and other parties affected 
by test-based decisions. The intent of the 
Standards is to promote the sound and ethical 
use of tests and to provide a basis for evaluat¬ 
ing the quality of testing practices. 

Participants in the Testing Process 

Educational and psychological testing and 
assessment involve and significantly affect 
individuals, institutions, and society as a 
whole. The individuals affected include stu¬ 
dents, parents, teachers, educational adminis¬ 
trators, job applicants, employees, clients, 
patients, supervisors, executives, and evalua¬ 
tors, among others. The institutions affected 
include schools, colleges, businesses, industry, 
clinics, and government agencies. Individuals 
and institutions benefit when testing helps them 
achieve their goals. Society, in turn, benefits 
when testing contributes to the achievement 
of individual and institutional goals. 

The interests of the various parties 
involved in the testing process are usually, 
but not always, congruent. For example, 
when a test is given for counseling purposes 
or for job placement, the interests of the 
individual and the institution often coin¬ 
cide. In contrast, when a test is used to 


select from among many individuals for a 
highly competitive job or for entry into an 
educational or training program, the prefer¬ 
ences of an applicant may be inconsistent 
with those of an employer or admissions 
officer. Similarly, when testing is mandated 
by a court, the interests of the test taker may 
be different from those of the party requesting 
the court order. 

There are many participants in the testing 
process, including, among others: (a) those who 
prepare and develop the test! (b) those who 
publish and market the test; (c) those who 
administer and score the test; (d) those who 
use the test results for some decision-making 
purpose; (e) those who interpret test results for 
clients; (f) those who take the test by choice, 
direction, or necessity; (g) those who sponsor 
tests, which may be boards that represent 
institutions or governmental agencies that 
contract with a test developer for a specific 
instrument or service; and (h) chose who select 
or review tests, evaluating their comparative, 
merits or suitability for the uses proposed. 

These roles are sometimes combined and 
sometimes further divided. For example, in 
clinics the test taker is typically the intended 
beneficiary of the test results. In some situa¬ 
tions the test administrator is an agent of the 
test developer, and sometimes the test admin¬ 
istrator is also the test user. When an industrial 
organization prepares its own employment 
tests, it is both the developer and the user. 
Sometimes a test is developed by a test author 
but published, advertised, and distributed by 
an independent publisher, though the publisher 
may play an active role in the test development. 
Given this intermingling of roles, it is difficult 
to assign precise responsibility for addressing 
various standards to specific participants in 
the testing process. 

This document begins with a series of 
chapters on the test development process, 
which focus primarily on the responsibilities 
of test developers, and then turns co chapters 
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on specific uses and applications, which focus 
primarily on responsibilities of test users. One 
chapter is devoted specifically to the rights 
and responsibilities of test takers. 

The Standards is based on the premise 
that effective testing and assessment require 
that all participants in the testing process pos¬ 
sess the knowledge, skills, and abilities rele¬ 
vant to their role in che testing process, as 
well as awareness of personal and contextual 
factors that may influence the testing process. 
They also should obtain any appropriate 
supervised experience and legislatively man¬ 
dated practice credentials necessary to perform 
competently those aspects of the testing 
process in which they engage. For example, 
test developers and those selecting and 
interpreting tests need adequate knowledge 
of psychometric principles such as validity 
and reliability. 

The Purpose of the Standards 

The purpose of publishing the Standards is 
to provide criteria for the evaluation of tests, 
testing practices, and the efFects of test use. 
Although the evaluation of the appropriate¬ 
ness of a test or testing application should 
depend heavily on professional judgment, the 
Standards provides a frame of reference to 
assure that relevant issues are addressed. It is 
hoped that all professional test developers, 
sponsors, publishers, and users will adopc the 
Standards and encourage others to do so. 

The Standards makes no attempt to pro¬ 
vide psychometric answers to questions of 
public policy regarding the use of tests. In 
general, the Standards advocates that, within 
feasible limits, the relevant technical informa¬ 
tion be made available so that those involved 
in policy debate may be fully informed. 

Categories of Standards 

The 1985 Standards designated each standard 
as “primary” (to be met by all tests before 
operational use), “secondary” (desirable, but 


not feasible in certain situations), or “condi¬ 
tional” (importance varies with application). 
The present Standards continues the tradition 
of expecting test developers and users to con¬ 
sider all standards before operational use; 
however, the Standards docs not continue the 
practice of designating levels of importance. 
Instead, the text of each standard, and any 
accompanying commentary, discusses the 
conditions under which a standard is relevant. 
It was not the case that under the 1985 
Standards test developers and users were obli¬ 
gated to attend only to the primary standards. 
Rather, the term “conditional” meant that a 
standard was primary in some settings and 
secondary in others, thus requiring careful 
consideration of the applicability of each stan¬ 
dard for a given setting. 

The absence of designations such as 
“primary” or "conditional" should not be 
taken to imply that all standards are equally 
significant in any given situation. Depending 
on the context and purpose of test develop¬ 
ment or use, some standards will be more 
salient than others. Moreover, some standards 
are broad in scope, setting forth concerns or 
requirements relevant to nearly all tests or 
testing contexts, and other standards are nar¬ 
rower in scope. However, all standards are 
important in the contexts to which they 
apply. Any classification that gives the appear¬ 
ance of elevating the general importance of 
some standards over others could invite neglect 
of some standards that need to be addressed 
in particular situations. 

Further, the current Standards does not 
include standards considered secondary or 
“desirable.” The continued use of che second¬ 
ary designation would risk encouraging both 
the expansion of the Standards to encompass 
large numbers of “desirable” standards and 
the inappropriate assumption that any guide¬ 
line not included in the Standards as at least 
“secondary" was inconsequential. 

Unless otherwise specified in the stan¬ 
dard or commentary, and with the caveats 
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outlined below, standards should be met 
before operational test use. This means that 
each standard should be carefully considered 
to determine its applicability to the resting 
context under consideration. In a given case 
there may be a sound professional reason why 
adherence to the standard is unnecessary. It is 
also possible rhar there may be occasions 
when technical feasibility may influence 
whether a standard can be met prior to 
operational test use. For example, some 
standards may call for analyses of data that 
may not be available at the point of initial 
operational test use. If test developers, users, 
and, when applicable, sponsors have deemed 
a standard to be inapplicable or unfeasible, 
they should be able, if called upon, to explain 
the basis for their decision. However, there 
is no expectation that documentation be 
routinely available of the decisions related 
to each standard. 

Tests and Test Uses to 
Which These Standards Apply 

A test is an evaluative device or procedure in 
which a sample of an examinee’s behavior in a 
specified domain is obtained and subsequent¬ 
ly evaluated and scored using a standardized 
process. While the label test is ordinarily 
reserved for instruments on which responses 
are evaluated for their correctness or quality 
and the terms scale or inventory are used for 
measures of attitudes, interest, and disposi¬ 
tions, the Standards uses the single term test 
to refer to all such evaluative devices. 

A distinction is sometimes made between 
test and assessment. Assessment is a broader 
term, commonly referring to a process that 
integrates test information with information 
from other sources (e.g., information from 
the individuals social, educational, employ¬ 
ment, or psychological history). The applica¬ 
bility of the Standards to an evaluation device 
or method is not altered by the label applied 
to it (e.g., test, assessment, scale, inventory). 


Tests differ on a number of dimensions: 
the mode in which test materials are present¬ 
ed (paper and pencil, oral, computerized 
administration, and so on); the degree to 
which stimulus materials are standardized; 
the type of response format (selection of a 
response from a set of alternatives as opposed 
to the production of a response); and the 
degree to which test materials are designed to 
reflect or simulate a particular context. In all 
cases, however, tests standardize the process 
by which test-taker responses to test materials 
are evaluated and scored. As noted in prior 
versions of the Standards, the same general 
types of information are needed for all vari¬ 
eties of tests. 

The precise demarcation between those 
measurement devices used in the fields of 
educational and psychological testing that do 
and do not fall within the purview of the 
Standards is difficult to identify. Although the 
Standards applies most directly to standard¬ 
ized measures generally recognized as "tests," 
such as measures of ability, aptitude, achieve¬ 
ment, attitudes, interests, personality, cogni¬ 
tive functioning, and mental health, it may 
also be usefully applied in varying degrees to 
a broad range of less formal assessment tech¬ 
niques. Admittedly, it will generally not be 
possible to apply the Standards rigorously to 
unstandardized questionnaires or to the broad 
range of unstructured behavior samples used 
in some forms of clinic- and school-based 
psychological assessment (e.g., an intake inter¬ 
view), and to instructor-made tests that are 
used to evaluate student performance in edu¬ 
cation and training. It is useful to distinguish 
between devices that lay claim to the concepts 
and techniques of the field of educational and 
psychological testing from those which repre¬ 
sent nonstandardized or less standardized aids 
to day-to-day evaluative decisions. Although 
the principles and concepts underlying the 
Standards can be fruitfully applied to day-to- 
day decisions, such as when a business owner 
interviews a job applicant, a manager evalu- 
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ates the performance of subordinates, or a 
coach evaluates a prospective athlete, it would 
be overreaching to expect that the standards 
of the educarional and psychological testing 
field be followed by those making such deci¬ 
sions. In contrast, a structured interviewing 
system developed by a psychologist and 
accompanied by claims that the system has 
been found to be predictive of job perform¬ 
ance in a variety of other settings falls within 
the purview of the Standards. 

Cautions to be Exercised in Using 
the Standards 

Several cautions are important to avoid mis¬ 
interpreting the Standards'. 

1) Evaluating the acceptability of a test 
or test application does not rest on the literal 
satisfaction of every standard in this docu¬ 
ment, and acceptability cannot be determined 
by using a checklist. Specific circumstances 
afFect the importance of individual standards, 
and individual standards should not be con¬ 
sidered in isolation. Therefore, evaluating 
acceptability involves (a) professional judgment 
that is based on a knowledge of behavioral sci¬ 
ence, psychometrics, and the community 
standards in the professional field to which 
che tescs apply; (b) the degree to which the 
intent of the standard has been satisfied by 
the test developer and user; (c) the alternatives 
that are readily available; and (d) research and 
experiential evidence regarding feasibility of 
meeting the standard. 

2) When tests are at issue in legal pro¬ 
ceedings and other venues requiring expert 
witness testimony it is essential that profes¬ 
sional judgment be based on the accepted 
corpus of knowledge in determining the rele¬ 
vance of particular standards in a given situa¬ 
tion. The intent of the Standards is to offer 
guidance for such judgments. 

3) Claims by test developers or test users 
that a test, manual, or procedure satisfies or 
follows these standards should be made with 


care. It is appropriate for developers or users 
to state that efforts were made to adhere to 
the Standards , and to provide documents 
describing and supporting those efforts. 
Blanket claims without supporting evidence 
should not be made. 

4) These standards are concerned with a 
field that is evolving. Consequently, there is 
a continuing need to monitor changes in the 
field and to revise this document as knowl¬ 
edge develops. 

5) Prescription of the use of specific 
technical methods is not the intent of the 
Standards. For example, where specific statis¬ 
tical reporting requirements are mentioned, 
the phrase “or generally accepted equivalent" 
always should be understood. 

The standards do not attempt to repeat 
ot to incorporate the many legal or regulatory 
requirements that might be relevant to the 
issues they address. In some areas, such as the 
collection, analysis, and use of test data and 
results for different subgroups, the law may 
both require participants in the testing process 
to take certain actions and prohibit those 
participants from taking other actions. Where 
it is apparent that one or more standards or 
comments address an issue on which estab¬ 
lished legal requirements may be particularly 
relevant, the standard, comment, or introduc¬ 
tory material may make note of that fact. 
Lack of specific reference to legal require¬ 
ments, however, does not imply that no rele¬ 
vant requirement exists. In all situations, 
participants in the testing process should 
separately consider and, where appropriate, 
obtain legal advice on legal and regulatory 
requirements. 

The Number of Standards 

The number of standards has increased from 
the 1985 Standards for a variety of reasons. 
First, and most importantly, new develop¬ 
ments have led to the addition of new stan¬ 
dards. Commonly these deal with new types 
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of tests or new uses for existing tests, rather 
than being broad standards applicable to all 
tests. Second, on the basis of recognition chat 
some users of the Standards may turn only to 
chapters directly relevant to a given applica¬ 
tion, certain standards are repeated in differ¬ 
ent chapters. When such repetition occurs, 
the essence of the standard is the same. Only 
the wording, area of application, or elabora¬ 
tion in the comment is changed. Third, 
standards dealing with important nontechni¬ 
cal issues, such as avoiding conflicts of inter¬ 
est and equitable treatment of ail test takers, 
have been added. Although such topics have 
not been addressed in prior versions of the 
Standards, they are not likely to be viewed as 
imposing burdensome new requirements. 
Thus che increase in the number of stan¬ 
dards does not per se signal an increase in 
the obligations placed on test developers 
and test users. 

Tests as Measures of Constructs 

We depart from some historical uses of the 
term “construct,” which reserve the term for 
characteristics that are not directly observable, 
but which are inferred from interrelated sets 
of observations. This historical perspective 
invites confusion. Some tests are viewed as 
measures of constructs, while others are not. 
In addition, considerable debate has ensued 
as to whether certain characteristics measured 
by tests are properly viewed as constructs. 
Furthermore, the types of validity evidence 
thought to be suitable can difFer as a result 
of whether a given test is viewed as measur¬ 
ing a construct. 

We use the term construct more broadly 
as the concept or characteristic that a test is 
designed to measure. Rarely, if ever, is there a 
single possible meaning that can be attached 
to a test score or a pattern of test responses. 
Thus, it is always incumbent on a testing 
professional to specify the construct interpre¬ 
tation that will be made on the basis of the 


score or response pattern. The notion that 
some tests are not under the purview of the 
Standards because they do not measure con¬ 
structs is contrary to this use of the term. 
Also, as detailed in chapter 1, evolving con¬ 
ceptualizations of the concept of validity no 
longer speak of different types of validity but 
speak instead of different iines of validity evi¬ 
dence, all in service of providing information 
relevant to a specific intended interpretation 
of test scores. Thus, many lines of evidence 
can contribute to an understanding of the 
construct meaning of test scores. 

Organization of This Volume 

Part I of the Standards , "Test Construction, 
Evaluation, and Documentation,” contains 
standards for validity (ch. 1); reliability and 
errors of measurement (ch. 2); test develop¬ 
ment and revision (ch. 3); scaling, norming, 
and score comparability (ch. 4); test adminis¬ 
tration, scoring, and reporting (ch. 5); and 
supporting documentation for tests (ch. 6). 
Part II addresses “Fairness in Testing,” and 
contains standards on fairness and bias (ch. 7); 
the rights and responsibilities of test takers 
(ch. 8); testing individuals of diverse linguis¬ 
tic backgrounds (ch. 9); and testing individu¬ 
als with disabilities (ch. 10). Part III treats 
specific "Testing Applications,” and contains 
standards involving general responsibilities of 
test users (ch. 11); psychological testing and 
assessment (ch. 12); educational testing and 
assessment (ch. 13); testing in employment 
and credentialing (ch. 14); and testing in pro¬ 
gram evaluation and public policy (ch. 15). 

Each chapter begins with introductory 
text that provides background for the stan¬ 
dards that follow. This revision of the 
Standards contains more extensive intro¬ 
ductory text material than its predecessor. 
Recognizing the common use of the Standards 
in the education of future test developers 
and users, the committee opted to provide a 
context for the standards themselves by pre- 
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senting more background material than in 
previous versions. This text is designed to 
assist in the interpretation of the standards 
that follow in each chapter. Although the text 
is at times prescriptive and exhortatory, it 
should not be interpreted as imposing addi¬ 
tional standards'. 

The Standards also contains an index and 
includes a glossary that provides definitions 
for terms as they are specifically used in this 
volume. 
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Background 

Validity refers co the degree to which evidence 
and theory support the interpretations of test 
scores entailed by proposed uses of tests. 
Validity is, therefore, the mosr fundamental 
consideration in developing and evaluating 
tests. The process of validation involves accu¬ 
mulating evidence to provide a sound scientific 
basis for the proposed score interpretations. 

Ir is the interpretations of rest scores required 
by proposed uses that are evaluated, not the 
test itself. When test scores are used or inter¬ 
preted in more than one way, each intended 
interpretation must be validated. 

Validation logically begins with an explicit 
statement of the proposed interpretation of 
test scores, along with a rationale for the rele¬ 
vance of the interpretation to the proposed 
use. The proposed interpretation refers to the 
construct or concepts the test is intended to 
measure. Examples of constructs are mathe¬ 
matics achievement, performance as a com¬ 
puter technician, depression, and self-esteem. 
To support test development, the proposed 
interpretation is elaborated by describing 
its scope and extent and by delineating the 
aspects of the construct that are to be repre¬ 
sented. The detailed description provides a 
conceptual framework for the test, delineat¬ 
ing the knowledge, skills, abilities, processes, 
or characteristics to be assessed. The frame¬ 
work indicates how this representation of 
the construct is to be distinguished from 
other constructs and how it should relate 
to other variables. 

The conceptual framework is partially 
shaped by the ways in which test scores will 
be used. For instance, a test of mathematics 
achievement might be used to place a student 
in an appropriate program of instruction, to 
endorse a high school diploma, or to inform 
a college admissions decision. Each of these 
uses implies a somewhat different interpre¬ 
tation of the mathematics achievement test 


scores: that a student will benefit from a 
particular instructional intervention, that a 
student has mastered a specified curriculum, 
or that a student is likely to be successful 
with college-level work. Similarly, a test of 
self-esteem might be used for psychological 
counseling, to inform a decision about 
employment, or for the basic scientific pur¬ 
pose of elaborating the construct of self-esteem. 
Each of these potential uses shapes die specified 
framework and the proposed interpretation of 
the test’s scores and also has implications for 
test development and evaluation. 

Validation can be viewed as developing a 
scientifically sound validity argument to sup¬ 
port the intended interpretation of test scores 
and their relevance to the proposed use. The 
conceptual framework points to the kinds of 
evidence that might be collected to evaluate 
the proposed interpretation in light of the 
purposes of testing. As validation proceeds, 
and new evidence about the meaning of a 
test’s scores becomes available, revisions may 
be needed in the test, in the conceptual 
framework that shapes it, and even in the 
construct underlying the test. 

The wide variety of tests and circum¬ 
stances makes it natural that some types of 
evidence will be especially cricical in a given 
case, whereas other types will be less useful. 
The decision about what types of evidence 
are important for validation in each instance 
can be clarified by developing a set of propo¬ 
sitions that support the proposed interpretation 
for the particular purpose of testing. For 
instance, when a mathematics achievement 
test is used to assess readiness for an advanced 
course, evidence for the following proposi¬ 
tions might be deemed necessary: (a) that cer¬ 
tain skills are prerequisite for the advanced 
course; (b) that the concent domain of the 
test is consistent with these prerequisite skills; 
(c) that test scores can be generalized across 
relevant sets of items; (d) that test scores are 
not unduly influenced by ancillary variables, 
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such as writing ability; (e) that success in the 
advanced course can be validly assessed; and 
(f) that examinees with high scores on the 
test will be more successful in the advanced 
course than examinees with low scores on the 
test. Examples of propositions in other testing 
contexts might include, for instance, the 
proposition that examinees with high general 
anxiety scores experience significant anxiety 
in a range of settings, the proposition that a 
child’s score on an intelligence scale is strong¬ 
ly related to the child’s academic performance, 
or the proposition that a certain pattern of 
scores on a neuropsychological battery indi¬ 
cates impairment characteristic of brain injury. 
The validation process evolves as these propo¬ 
sitions are articulated and evidence is gathered 
to evaluate their soundness. 

Identifying the propositions implied by 
a proposed test interpretation can be facili¬ 
tated by considering rival hypotheses that 
may challenge the proposed interpretation. 
It is also useful to consider the perspectives 
of different interested parties, existing expe¬ 
rience with similar tests and contexts, and 
the expected consequences of the proposed 
test use. Plausible rival hypotheses can often 
be generated by considering whether a test 
measures less or more than its proposed 
construct. Such concerns are referred to as 
construct underrepresentation and construct- 
irrelevant variance. 

Construct underrepresentation refers to 
the degree to which a test fails to capture 
important aspects of the construct. It implies 
a narrowed meaning of test scores because 
the test does not adequately sample some 
types of content, engage some psychological 
processes, ot elicit some ways of responding 
that are encompassed by the intended con¬ 
struct. Take, for example, a test of reading 
comprehension intended to measure chil¬ 
dren’s ability to read and interpret stories 
with understanding. A particular test might 
underrepresent the intended construct because 
it did not contain a sufficient variety of read¬ 


ing passages or ignored a common type of 
reading material. As another example, a test 
of anxiety might measure only physiological 
reactions and not emotional, cognitive, or 
situational components. 

Construct-irrelevant variance refers to 
the degree to which test scores are affected by 
processes that are extraneous to its intended 
construct. The test scores may be systemati¬ 
cally influenced to some extent by compo¬ 
nents that are not part of the construct. In 
the case of a reading comprehension test, 
construct-irrelevant components might 
include an emotional reaction to the cest 
content, familiarity with the subject matter 
of the reading passages on the test, or the 
writing skill needed to compose a response. 
Depending on the detailed definition of the 
construct, vocabulary knowledge or reading 
speed might also be irrelevant components. 
On a test of anxiety, a response bias to under¬ 
report anxiety might be considered a source 
of construct-irrelevant variance. 

Nearly all tests leave out elements that 
some potential users believe should be meas¬ 
ured and include some elements that some 
potential users consider inappropriate. 
Validation involves careful attention to possible 
distortions in meaning arising from inadequate 
representation of the construct and also to 
aspeccs of measurement such as test format, 
administration conditions, or language level 
that may materially limit or qualify the inter¬ 
pretation of test scores. That is, the process 
of validation may lead to revisions in the test, 
the conceptual framework of the test, or both. 
The revised test would then need validation. 

When propositions have been identified 
that would support the proposed interpretation 
of test scores, validation can proceed by devel¬ 
oping empirical evidence, examining relevant 
literature, and/or conducting logical analyses to 
evaluate each of these propositions. Empirical 
evidence may include both local evidence, 
produced within the contexts where the test 
will be used, and evidence from similar testing 
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applications in other settings. Use of existing 
evidence from similar tests and contexts can 
enhance the quality of the validity argument, 
especially when current data are limited. 

Because a validity argument typically 
depends on more than one proposition, strong 
evidence in support of one in no way dimin¬ 
ishes the need for evidence to support others. 
For example, a strong predictor-criterion rela¬ 
tionship in an employment setting is not suf¬ 
ficient to justify test use for selection without 
considering the appropriateness and meaning¬ 
fulness of the criterion measure. Professional 
judgment guides decisions regarding the spe¬ 
cific forms of evidence that can best support 
the intended interpretation and use. As in 
all scientific endeavors, the quality of the 
evidence is primary. A few lines of solid evi¬ 
dence regarding a particular proposition are 
better than numerous lines of evidence of 
questionable quality. 

Validation is the joint responsibility of 
the test developer and the test user. The test 
developer is responsible for furnishing rele¬ 
vant evidence and a rationale in support of 
the intended test use. The test user is ultimately 
responsible for evaluating the evidence in the 
particular setting in which the test is to be 
used. When the use of a test differs from that 
supported by the test developer, the test user 
bears special responsibility for validation. The 
standards apply to the validation process, for 
which the appropriate parties share responsi¬ 
bility. It should be noted that important con¬ 
tributions to the validity evidence are made as 
other researchers report findings of investiga¬ 
tions that are related to the meaning of scores 

Sources of Validity Evidence 

The following sections outline various sources 
of evidence that might be used in evaluating a 
proposed interpretation of test scores for par¬ 
ticular purposes. These sources of evidence 
may illuminate different aspects of validity, 


but they do not represent distinct types of 
validity. Validity is a unitary concept. It is the 
degree to which all the accumulated evidence 
supports the intended interpretation of test 
scores for the proposed purpose. Like the 
1985 Standards, this edition refers to types of 
validity evidence, rather than distinct types of 
validity. To emphasize this distinction, the 
treatment that follows does not follow tradi¬ 
tional nomenclature (i.e., the use of the terms 
content validity or predictive validity). The 
glossary contains definitions of the traditional 
terms, explicating the difference between tra¬ 
ditional and current use. 

Evidence Based on Test Content 

Important validity evidence can be obtained 
from an analysis of the relationship between a 
test’s content and the construct it is intended 
to measure. Test content refers to the themes, 
wording, and format of the items, tasks, or 
quesdons on a test, as well as the guidelines for 
procedures regarding administration and scor¬ 
ing. Test developers often work from a specifi¬ 
cation of the content domain. The content 
specification carefully describes the content in 
detail, often with a classification of areas of 
content and types of items. Evidence based on 
test content can include logical or empirical 
analyses of the adequacy with which the test 
content represents the content domain and of 
the relevance of the content domain to the 
proposed interpretation of test scores. Evidence 
based on content can also come from expert 
judgments of the relationship between parts 
of the test and the construct. For example, in 
developing a licensure test, the major facets of 
the specific occupation can be specified, and 
experts in that occupation can be asked to 
assign test items to the categories defined by 
those facets. They, or other qualified experts, 
can then judge che representativeness of the 
chosen set of items. Sometimes rules or algo¬ 
rithms can be constructed to select or generate 
items that differ systematically on the various 
facets of content, according to specifications. 
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Some tests are based on systematic obser¬ 
vations of behavior. For example, a listing of 
the tasks comprising a job domain may be 
developed from observations of behavior in a 
job, together with judgments of subject-matter 
experts. Expert judgments can be used to assess 
the relative importance, criticality, and/or fre¬ 
quency of the various tasks. A job sample test 
can then be constructed from a random or 
stratified sampling of casks rated highly on 
these characteristics. The test can then be 
administered under standardized conditions 
in an off-the-job setting. 

The appropriateness of a given concent 
domain is related to the specific inferences to 
be made from tesc scores. Thus, when consid¬ 
ering an available test for a purpose ocher chan 
that for which it was first developed, it is 
especially important to evaluate the appropri¬ 
ateness of the original content domain for the 
proposed new use. In educational program 
evaluations, for example, tests may properly 
cover material thac receives little or no atten¬ 
tion in the curriculum, as well as that toward 
which instruction is directed. Policymakers 
can then evaluate student achievement with 
respect to both content neglected and content 
addressed. On the other hand, when student 
mastery of a delivered curriculum is tested for 
purposes of informing decisions about indi¬ 
vidual students, such as promotion or gradua¬ 
tion, the framework elaborating a content 
domain is appropriately limited to what stu¬ 
dents have had an opportunity to learn from 
the curriculum as delivered. 

Evidence about content can be used, in 
part, to address questions about differences in 
the meaning or interpretation of test scores 
across relevant subgroups of examinees. Of 
particular concern is the extent to which con¬ 
struct underrepresentation or construct-irrele¬ 
vant components may give an unfair advantage 
or disadvantage to one or more subgroups of 
examinees. Careful review of the construct 
and test content domain by a diverse panel 
of experts may point co potential sources of 


irrelevant difficulty (or easiness) that require 
further investigation. 

Evidence Based on Response Processes 

Theoretical and empirical analyses of the 
response processes of test takers can provide 
evidence concerning the fit between the con¬ 
struct and the detailed nature of performance 
or response actually engaged in by examinees. 
For instance, if a tesr is intended to assess 
mathematical reasoning, it becomes impor¬ 
tant to determine whether examinees are, in 
fact, reasoning about the material given instead 
of following a standard algorithm. For another 
instance, scores on a scale intended to assess 
the degree of an individual’s extroversion or 
introversion should not be strongly influenced 
by social conformity. 

Evidence based on response processes 
generally comes from analyses of individual 
responses. Questioning test takers about their 
performance strategies or responses to partic¬ 
ular items can yield evidence that enriches the 
definition of a construct. Maintaining records 
that monitor the development of a response 
to a writing task, through successive written 
drafts or electronically monitored revisions, 
for instance, also provides evidence of process. 
Documentation of other aspects of performance, 
like eye movements or response times, may 
also be relevant to some constructs. Inferences 
about processes involved in performance can 
also be developed by analyzing the relationship 
among parrs of the test and between the test 
and other variables. Wide individual differ¬ 
ences in process can be revealing and may lead 
to reconsideration of certain test formats. 

Evidence of response processes can 
contribute to questions about differences in 
meaning or interpretation of test scores across 
relevant subgroups of examinees. Process stud¬ 
ies involving examinees from different sub¬ 
groups can assist in determining the extent to 
which capabilities irrelevant or ancillary to the 
construct may be differentially influencing 
their performance. 
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Studies of response processes are not lim¬ 
ited to the examinee. Assessments often rely 
on observers or judges to record and/or evalu¬ 
ate examinees’ performances or products. In 
such cases, relevant validity evidence includes 
the extent to which the processes of observers 
or judges are consistent with the intended 
interpretation of scores. For instance, if 
judges are expected to apply particular criteria 
in scoring examinees’ performances, it is 
important to ascertain whether they are, in 
fact, applying the appropriate criteria and not 
being influenced by factors that are irrelevant 
to the intended interpretation. Thus, valida¬ 
tion may include empirical studies of how 
observers or judges record and evaluate data 
along with analyses of the appropriateness of 
these processes to the intended interpretation 
or construct definition. 

Evidence Based on Internal Structure 

Analyses of the internal structure of a 
test can indicate the degree to which the 
relationships among test items and test com¬ 
ponents conform to the construct on which 
the proposed test score interpretations are 
based. The conceptual framework for a test 
may imply a single dimension of behavior, 
or it may posit several components that are 
each expected to be homogeneous, but that 
are also distinct from each other. For exam¬ 
ple, a measure of discomfort on a health sur¬ 
vey might assess both physical and emotional 
health. The extent to which item interrela¬ 
tionships bear out the presumptions of the 
framework would be relevant to validity. 

The specific types of analysis and their 
interpretation depend on how the test will 
be used. For example, if a particular appli¬ 
cation posited a series of test components of 
increasing difficulty, empirical evidence of 
the extent to which response patterns con¬ 
formed to this expectation would be provid¬ 
ed. A theory that posited unidimensionality 
would call for evidence of item homogene¬ 
ity. In this case, the item interrelationships 


also provide an estimate of score reliability, 
but such an index would be inappropriate for 
tests with a more complex internal structure. 

Some studies of the internal structure of 
tests are designed to show whether particular 
items may function differently for identifiable 
subgroups of examinees. Differential item 
functioning occurs when different groups 
of examinees with similar overall ability, or 
similar status on an appropriate criterion, 
have, on average, systematically different 
responses to a particular item. This issue is 
discussed in chapters 3 and 7. However, dif¬ 
ferential item functioning is not always a 
flaw or weakness. Subsets of items that have 
a specific characteristic in common (e.g., 
specific content, task representation) may 
function differently for different groups of 
similarly scoring examinees. This indicates 
a kind of multidimensionality that may be 
unexpected or may conform to the test 
framework. 

Evidence Based on Relations to Other Variables 

Analyses of the relationship of test scores 
to variables external to the test provide anoth¬ 
er important source of validity evidence. 
External variables may include measures of 
some criteria that the test is expected to pre¬ 
dict, as well as relationships to other tests 
hypothesized to measure the same constructs, 
and tests measuring related or different con¬ 
structs. Measures other than test scores, such 
as performance criteria, are often used in 
employment settings. Categorical variables, 
including group membership variables, 
become relevant when the theory underlying 
a proposed test use suggests that group differ¬ 
ences should be present or absent if a pro¬ 
posed test interpretation is to be supported. 
Evidence based on relationships with other 
variables addresses questions about the degree 
to which these relationships are consistent 
with the construct underlying the proposed 
test interpretations. 
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Convergent and discriminant evidence. 
Relationships between test scores and other 
measures intended to assess similar constructs 
provide convergent evidence, whereas rela¬ 
tionships between tesc scores and measures 
purportedly of different constructs provide 
discriminant evidence. For instance, within 
some theoretical frameworks, scores on a 
multiple-choice test of reading comprehen¬ 
sion might be expected ro relate closely 
(convergent evidence) to other measures of 
reading comprehension based on other meth¬ 
ods, such as essay responses; conversely, test 
scores might be expected to relate less closely 
(discriminant evidence) to measures of other 
skills, such as logical reasoning. Relationships 
among different methods of measuring the 
construct can be especially helpful in sharp¬ 
ening and elaborating score meaning and 
interpretation. 

Evidence of relations with other variables 
can involve experimental as well as correla¬ 
tional evidence. Studies might be designed, 
for instance, to investigate whether scores on 
a measure of anxiety improve as a result of 
some psychological treatment or whether 
scores on a test of academic achievement dif¬ 
ferentiate between instructed and nonin- 
structed groups. If performance increases due 
to short-term coaching are viewed as a threat 
to validity, it would be useful to investigate 
whether coached and uncoached groups per¬ 
form differently. 

Test-criterion relationships. Evidence of 
the relation of test scores to a relevant criterion 
may be expressed in various ways, but the 
fundamental question is always: How accu¬ 
rately do test scores predict criterion per¬ 
formance? The degree of accuracy deemed 
necessary depends on the purpose for which 
the test is used. 

The criterion variable is a measure of some 
attribute or outcome that is of primary inter¬ 
est, as determined by test users, who may be 
administrators in a school system, the man¬ 
agement of a firm, or clients. The choice of 

14 


the criterion and the measurement procedures 
used to obtain criterion scores are of central 
importance. The value of a test-criterion study 
depends on the relevance, reliability, and validity 
of the interpretation based on the criterion 
measure for a given testing application. 

Historically, two designs, often called 
predictive and concurrent, have been distin¬ 
guished for evaluating test-criterion relation¬ 
ships. A predictive study indicates how 
accurately test data can predict criterion scores 
that are obtained at a later time. A concurrent 
study obtains predictor and criterion infor¬ 
mation at about the same time. When predic¬ 
tion is actually contemplated, as in education 
or employment settings, or in planning reha¬ 
bilitation regimens, predictive studies can 
recain the temporal differences and other 
characteristics of the practical situation. 
Concurrent evidence, which avoids temporal 
changes, is particularly useful for psychodiag¬ 
nostic tests or to investigate alternative meas¬ 
ures of some specified construct. In general, 
the choice of research strategy is guided by 
prior evidence of the extent to which predic¬ 
tive and concurrent studies yield the same or 
different results in the domain. 

Test scores are sometimes used in allocat¬ 
ing individuals to different treatments, such as 
different jobs within an institution, in a way 
that is advantageous for the institution and for 
the individuals. In that context, evidence is 
needed to judge the suitability of using a test 
when classifying or assigning a person to one 
job versus another or to one treatment versus 
another. Classification decisions are supported 
by evidence that the relationship of test scores 
to performance criteria is different for different 
treatments. It is possible for tests to be highly 
predictive of performance for different educa¬ 
tion programs or jobs without providing the 
information necessary to make a comparative 
judgment of the efficacy of assignments or 
treatments. In general, decision rules for 
selection or placement are also influenced by 
the number of persons to be accepted or the 
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numbers that can be accommodated in alter¬ 
native placement categories. 

Evidence about relations to other vari¬ 
ables is also used to investigate questions of 
differential prediction for groups. For instance, 
a finding that the relation of test scores to a 
relevant criterion variable differs from one 
group to another may imply that the mean¬ 
ing of the scores is not the same for members 
of the different groups, perhaps due to con¬ 
struct underrepresentation or construct-irrele¬ 
vant components. However, the difference 
may also imply that the criterion has different 
meaning for different groups. The differences 
in test-criterion relationships can also arise 
from measurement error, especially when 
group means differ, so such differences do 
not necessarily indicate differences in score 
meaning. (See chapter 7.) 

Validity generalization. An important 
issue in educational and employment settings 
is the degree to which evidence of validity 
based on test-criterion relations can be gener¬ 
alized to a new situation without further study 
of validity in that new situation. When a test 
is used to predict the same or similar criteria 
(e.g., performance of a given job) at different 
times or in different places, it is typically found 
that observed test-criterion correlations vary 
substantially. In the past, this has been taken 
to imply that local validation studies are always 
required. Mote recendy, meta-analytic analyses 
have shown that in some domains, much of 
chis variability may be due to statistical artifacts 
such as sampling fluctuations and variations 
across validation studies in the ranges of test 
scores and in the reliability of criterion meas¬ 
ures. When these and ocher influences are taken 
into account, it may be found that the remain¬ 
ing variability in validity coefficients is relatively 
small. Thus, statistical summaries of past vali¬ 
dation studies in similar situations may be 
useful in estimating test-criterion relationships 
in a new situation. This practice is referred to 
as the study of validity generalization. 


In some circumstances, there is a strong 
basis for using validity generalization. This 
would be the case where the meta-analytic 
database is large, where the meta-analytic data 
adequately represent the type of situation to 
which one wishes to generalize, and where 
correction for statistical artifacts produces a 
clear and consistent pattern of validity evi¬ 
dence. In such circumstances, the informa¬ 
tional value of a local validity study may be 
relatively limited. In other circumstances, the 
inferential leap required for generalization 
may be much larger. The meta-analytic data¬ 
base may be small, the findings may be less 
consistent, or the new situation may involve 
features markedly different from those repre¬ 
sented in the meta-analytic database. In such 
circumstances, situation-specific evidence of 
validity will be relatively more informative. 
Although research on validity generalization 
shows that results of a single local validation 
study may be quite imprecise, there are situa¬ 
tions where a single study, carefully done, 
with adequate sample size, provides sufficient 
evidence to support test use in a new situa¬ 
tion. This highlights the importance of exam¬ 
ining carefully the comparative informational 
value of local versus meta-analytic studies. 

In conducting studies of the generaiiz- 
ability of validity evidence, the prior studies 
that are included may vary according to sev¬ 
eral situational facets. Some of the major 
facets are (a) differences in the way the pre¬ 
dictor construct is measured, (b) the type of 
job or curriculum involved, (c) the type of 
criterion measure used, (d) the type of test 
takers, and (e) the time period in which the 
study was conducted. In any particular study 
of validity generalization, any number of these 
facets might vary, and a major objective of the 
study is to determine empirically the extent 
to which variation in these facets affects the 
test-criterion correlations obtained. 

The extent to which predictive or con¬ 
current evidence of validity generalization can 
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be used in new situations is in large measure 
a function of accumulated research. Although 
evidence of generalization can often help to 
support a claim of validity in a new situation, 
the extent of available data limits the extent to 
which the claim can be sustained. 

The above discussion focuses on the use 
of cumulative databases to estimate predictor- 
criterion relationships. Meta-analytic tech¬ 
niques can also be used to summarize other 
forms of data relevant to other inferences one 
may wish to draw from test scores in a partic¬ 
ular application, such as effects of coaching 
and effects of certain alterations in testing 
conditions to accommodate test takers with 
certain disabilities. 

Evidence Based on Consequences of Testing 

An issue receiving attention in recent 
years is the incorporation of the intended and 
unintended consequences of test use into the 
concept of validity. Evidence about conse¬ 
quences can inform validity decisions. Here, 
however, it is important to distinguish 
between evidence that is directly relevant to 
validity and evidence that may inform deci¬ 
sions about social policy but falls outside 
the realm of validity. 

Distinguishing between issues of validity 
and issues of social policy becomes particularly 
important in cases where differential conse¬ 
quences of test use are observed for different 
identifiable groups. For example, concerns 
have been raised about the effect of group 
differences in test scores on employment 
selection and promotion, the placement of 
children in special education classes, and the 
narrowing of a school’s curriculum to exclude 
learning of objectives that are not assessed. 
Although information about the consequences 
of resting may influence decisions about test 
use, such consequences do not in and of 
themselves detract from the validity of intended 
test interpretations. Rather, judgments of 
validity or invalidity in the light of testing 


consequences depend on a more searching 
inquiry into the sources of those consequences. 

Take, as an example, a finding of different 
hiring rates for members of different groups as 
a consequence of using an employment test. If 
the difference is due solely to an unequal distri¬ 
bution of the skills the test purports to meas¬ 
ure, and if those skills are, in fact, important 
contributors to job performance, then the find¬ 
ing of group differences per se does not imply 
any lack of validity for the intended inference. 
If, however, the test measured skill differences 
unrelated to job performance (e.g., a sophisti¬ 
cated reading test for a job that required only 
minimal functional literacy), or if the differ¬ 
ences were due to the test’s sensitivity to some 
examinee characteristic not intended to be part 
of the test construct, then validity would be 
called into question, even if test scores correlat¬ 
ed positively with some measure of job per¬ 
formance. Thus, evidence about consequences 
may be directly relevant to validity when it can 
be traced to a source of invalidity such as con¬ 
struct underrepresentation or construct-irrele¬ 
vant components. Evidence about consequences 
that cannot be so traced—that in fact reflects 
valid differences in performance—is crucial in 
informing policy decisions but falls outside the 
technical purview of validity. 

Tests are commonly administered in the 
expectation that some benefit will be realized 
from the intended use of the scores. A few of 
the many possible benefits are selection of 
efficacious treatments for therapy, placement 
of workers in suitable jobs, prevention of 
unqualified individuals from entering a pro¬ 
fession, or improvement of classroom instruc¬ 
tional practices. A fundamental purpose of 
validation is to indicate whether these specific 
benefits are likely to be realized. Thus, in the 
case of a test used in placement decisions, the 
validation would be informed by evidence 
that alternative placements, in fact, are dif¬ 
ferentially beneficial co the persons and the 
institution. In the case of employment testing. 
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standards! 


if a test publisher claims that use of the test 
will result in reduced employee training costs, 
improved workforce efficiency, or some other 
benefit, then the validation would be informed 
by evidence in support of that claim. 

Claims are sometimes made for benefits 
of testing that go beyond direct uses of the 
test scores themselves. Educational tests, for 
example, may be advocated on the grounds 
that their use will improve student motiva¬ 
tion or encourage changes in classroom 
instructional practices by holding educators 
accountable for valued learning outcomes. 
Where such claims are central to the rationale 
advanced for testing, the direct examination 
of testing consequences necessarily assumes 
even greater importance. The validation 
process in such cases would be informed by 
evidence that the anticipated benefits of test¬ 
ing are being realized. 

Integrating the Validity Evidence 

A sound validity argument integrates various 
strands of evidence into a coherent account 
of the degree to which existing evidence and 
theory support the intended interpretation of 
test scores for specific uses. It encompasses 
evidence gathered from new studies and evi¬ 
dence available from earlier reported research. 
The validity argument may indicate the need 
for refining the definition of the construct, may 
suggest revisions in the test or other aspects 
of the testing process, and may indicate areas 
needing further study. 

Ultimately, the validity of an intended 
interpretation of test scores relies on all the 
available evidence relevant to the technical 
quality of a testing system. This includes evi¬ 
dence of careful test construction; adequate 
score reliability; appropriate test administration 
and scoring; accurate score scaling, equating, 
and standard setting; and careful attention to 
fairness for all examinees, as described in subse¬ 
quent chapters of the Standards. 


Standard 1.1 

A rationale should be presented for each rec¬ 
ommended interpretation and use of test 
scores, together with a comprehensive sum¬ 
mary of the evidence and theory bearing on 
the intended use or interpretation. 

Comment: The rationale should indicate what 
propositions are necessary to investigate the 
intended interpretation. The comprehensive 
summary should combine logical analysis 
with empirical evidence to provide support 
for the test rationale. Evidence may come 
from studies conducted locally, in the setting 
where the test is to be used; from specific 
prior studies; or from comprehensive statisti¬ 
cal syntheses of available studies meeting 
clearly specified criteria. No type of evidence 
is inherently preferable to others; rather, the 
quality and relevance of the evidence to the 
intended test use determine the value of a 
particular kind of evidence. A presentation of 
empirical evidence on any point should give 
due weight to all relevant findings in the sci¬ 
entific literature, including those inconsistent 
with the intended interpretation or use. Test 
developers have the responsibility to provide 
support for their own recommendations, but 
test users are responsible for evaluating the 
quality of the validity evidence provided and 
its relevance to the local situation. 

Standard 1.2 

The test developer should set forth clearly 
how test scores are intended to be interpret¬ 
ed and used. The population(s) for which a 
test is appropriate should be clearly delimit¬ 
ed, and the construct that the test is intend¬ 
ed to assess should be clearly described. 

Comment: Statements about validity should 
refer to particular interpretations and uses. It 
is incorrect to use the unqualified phrase “the 
validity of the cesc." No test is valid for all 
purposes or in all situations. Each recom- 
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mended use or interpretation requires valida¬ 
tion and should specify in clear language the 
population for which the test is intended, the 
construct it is intended to measure, and the 
manner and contexts in which test scores are 
to" be employed. 

Standard 1.3 

If validity for some common or likely inter¬ 
pretation has not been investigated, or if the 
interpretation is inconsistent with available 
evidence, that fact should be made clear and 
potential users should be cautioned about 
making unsupported interpretations. 

Comment: If past experience suggests that a 
test is likely to be used inappropriately for 
certain kinds of decisions, specific warnings 
against such uses should be given. On the 
other hand, no two situations are ever identi¬ 
cal, so some generalization by the user is 
always necessary. Professional judgment is 
requited to evaluate the extent to which exist¬ 
ing validity evidence supports a given test use. 

Standard 1.4 

If a test is used in a way that has not been 
validated, it is incumbent on the user to jus¬ 
tify the new use, collecting new evidence if 
necessary. 

Comment: Professional judgment is required to 
evaluate the extent to which existing validity 
evidence applies in the new situation and to 
determine what new evidence may be needed. 
The amount and kinds of new evidence 
required may be influenced by experience with 
similar prior test uses or interpretations and 
by the amount, quality, and relevance of 

Standard 1.5 

The composition of any sample of exam¬ 
inees from which validity evidence is 


obtained should be described in as much 
detail as is practical, including major rele¬ 
vant sociodemographic and developmental 
characteristics. 

Comment: Statistical findings can be influ¬ 
enced by factors affecting the sample on 
which the results are based. When the sample 
is intended to represent a population, that 
population should be described, and atten¬ 
tion should be drawn to any systematic fac¬ 
tors that may limit the representativeness of 
the sample. Factors that might reasonably be 
expected to affect the results include self¬ 
selection, attrition, linguistic prowess, disabil¬ 
ity status, and exclusion criteria, and others. 

If the subjects of a validity study are patients, 
for example, then the diagnoses of the 
patients are important, as well as other char¬ 
acteristics, such as the severity of the diag¬ 
nosed condition. For tests used in industry, 
the employment status (e.g., applicants versus 
current job holders), the general level of expe¬ 
rience and educational background and the 
gender and ethnic composition of the sample 
may be relevant information. For tests used 
in educational settings, relevant information 
may include educational background, devel¬ 
opmental level, community characteristics, or 
school admissions policies, as well as the gen¬ 
der and ethnic composition of the sample. 
Sometimes restrictions about privacy preclude 
obtaining such population information. 

Standard 1.6 

When the validation rests in part on the 
appropriateness of test content, the procedures 
followed in specifying and generating test con¬ 
tent should be described and justified in refer¬ 
ence to the construct the test is intended to 
measure or the domain it is intended to repre¬ 
sent. If the definition of the content sampled 
incorporates criteria such as importance, fre¬ 
quency, or criticality, these criteria should also 
be clearly explained and justified. 
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Comment: For example, test developers might 
provide a logical structure that maps rhe 
items on the test to the content domain, 
illustrating the relevance of each item and the 
adequacy with which the set of items repre¬ 
sents the content domain. Areas of the content 
domain that are not included among the test 
items could be indicated as well. 

Standard 1.7 

When a validation rests in part on the opin¬ 
ions or decisions of expert judges, observers, 
or raters, procedures for selecting such 
experts and for eliciting judgments or rat¬ 
ings should be fully described. The qualifi¬ 
cations, and experience, of the judges should 
be presented. The description of procedures 
should include any training and instructions 
provided, should indicate whether partici¬ 
pants reached their decisions independently, 
and should report the level of agreement 
reached. If participants interacted with one 
another or exchanged information, the pro¬ 
cedures through which they may have influ¬ 
enced one another should be set forth. 

Comment: Systematic collection of judgments 
or opinions may occur at many points in cest 
construction (e.g., in eliciting expert judg¬ 
ments of content appropriateness or adequate 
content representation), in formulating rules 
or standards for score interpretation (e.g., in 
setting cut scores), or in test scoring (e.g., rat¬ 
ing of essay responses). Whenever such proce¬ 
dures are employed, the quality of the resulting 
judgments is important to the validation, ft 
may be entirely appropriate to have experts 
work together to reach consensus, but it would 
not then be appropriate to treat their respective 
judgments as statistically independent. 

Standard 1.8 

If the rationale for a test use or score inter¬ 
pretation depends on premises about the 
psychological processes or cognitive opera¬ 


tions used by examinees, then theoretical or 
empirical evidence in support of those prem¬ 
ises should be provided. When statements 
about the processes employed by observers 
or scorers are part of the argument for valid¬ 
ity, similar information should be provided. 

Comment: If the test specification delineates 
the processes to be assessed, then evidence is 
needed that the test items do, in feet, tap the 
intended processes. 

Standard 1.9 

If a test is claimed to be essentially unaffect¬ 
ed by practice and coaching, then the sensi¬ 
tivity of test performance to change with 
these forms of instruction should be docu¬ 
mented. 

Comment: Materials to aid in score interpreta¬ 
tion should summarize evidence indicating 
the degree to which improvement with prac¬ 
tice or coaching can be expected. Also, materi¬ 
als writcen for test takers should provide 
practical guidance about the value of test 
preparation activities, including coaching. 

Standard 1.10 

When interpretation of performance on spe¬ 
cific items, or small subsets of items, is sug¬ 
gested, the rationale and relevant evidence in 
support of such interpretation should be 
provided. When interpretation of individual 
item responses is likely but is not recom¬ 
mended by the developer, the user should be 
warned against making such interpretations. 

Comment: Users should be given sufficient 
guidance to enable them to judge the degree 
of confidence warranted for any use or inter¬ 
pretation recommended by the test developer. 
Test manuals and score reports should dis¬ 
courage overinterpretation of information 
that may be subject to considerable error. 
This is especially important if interpretation 
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of performance on isolated items, small sub¬ 
sets of items, or subtest scores is suggested. 

Standard 1.11 

If the rationale for a test use or interpreta¬ 
tion depends on premises about the relation¬ 
ships among parts of the test, evidence 
concerning the interna! structure of the test 
should be provided. 

Comment: It might be claimed, for example, 
that a test is essentially unidimensional. 
Such a claim could be supported by a mul¬ 
tivariate statistical analysis, such as a factor 
analysis, showing that the score variability 
attributable to one major dimension was 
much greater than the score variability 
attributable to any other identified dimen¬ 
sion. When a test provides more than one 
score, the interrelationships of those scores 
should be shown to be consistent with the 
construct(s) being assessed. 

Standard 1.12 

'When interpretation of subscores, score dif¬ 
ferences, or profiles is suggested, the ration¬ 
ale and relevant evidence in support of such 
interpretation should be provided. Where 
composite scores are developed, the basis 
and rationale for arriving at the composites 
should be given. 

Comment: When a test provides more than 
one score, the distinctiveness of the separate 
scores should be demonstrated, and the inter¬ 
relationships of those scores should be shown 
to be consistent with the construct(s) being 
assessed. Moreover, evidence for the validity 
of interpretations of two separate scores would 
not necessarily justify an interpretation of the 
difference between them. Rather, the rationale 
and supporting evidence must pertain directly 
to the specific score or score combination to 
be interpreted or used. 
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Standard 1.13 

When validity evidence includes statistical 
analyses of test results, either alone or 
together with data on other variables, the 
conditions under which the data were col¬ 
lected should be described in enough detail 
that users can judge the relevance of the 
statistical findings to local conditions. 
Attention should be drawn to any features 
of a validation data collection that are likely 
to differ from typical operational testing 
conditions and that could plausibly influ¬ 
ence test performance. 

Comment: Such conditions might include 
(but would not be limited to) the following: 
examinee motivation or prior preparation, the 
distribution of test scores over examinees, the 
time allowed for examinees to respond or 
ocher administrative conditions, examiner 
training or other examiner characteristics, 
the time intervals separating collection of 
data on different measures, or conditions 
that may have changed since the validity 
evidence was obtained. 

Standard 1.14 

When validity evidence includes empirical 
analyses of test responses together with data 
on other variables, the rationale for selecting 
the additional variables should be provided. 
Where appropriate and feasible, evidence 
concerning the constructs represented by 
other variables, as well as their technical 
properties, should be presented or cited. 
Attention should be drawn to any likely 
sources of dependence (or lack of independ¬ 
ence) among variables other than dependen¬ 
cies among the construct(s) they represent. 

Comment: The patterns of association 
between and among scores on the instrument 
under study and other variables should be 
consistent with theoretical expectations. The 
additional variables might be demographic 
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characteristics, indicators of treatment condi¬ 
tions, or scores on other measures. They 
might include intended measures of the same 
construct or of different constructs. The relia¬ 
bility of scores from such other measures and 
the validity' of intended interpretations of • 
scores from these measures are an important 
part of the validity evidence for the instru¬ 
ment under study. If such variables include 
composite scores, the construction of the 
composites should be explained. In addition 
to considering the properties of each variable 
in isolation, it is important to guard against 
faulty interpretations arising from spurious 
sources of dependency among measures, 
including correlated errors or shared variance 
due to common methods of measurement or 
common elements. 

Standard 1.15 

When it is asserted that a certain level of 
test performance predicts adequate or 
inadequate criterion performance, informa¬ 
tion about the levels of criterion perform¬ 
ance associated with given levels of test 
scores should be provided. 

Comment: Regression equations are more use¬ 
ful than correlation coefficients, which are 
generally insufficient to fully describe patterns 
of association between tests and other vari¬ 
ables. Means, standard deviations, and other 
statistical summaries are needed, as well as 
information about the distribution of criteri¬ 
on performances conditional upon a given 
test score. Evidence of overall association 
becween variables should be supplemented by 
information about the form of that associa¬ 
tion and about the variability associated with 
that association in different ranges of test 
scores. Note that data collections employing 
examinees selected for their extreme scores on 
one or more measures (extreme groups) typi¬ 
cally cannot provide adequate information 
about the association. 


Standard 1.16 

When validation relies on evidence that test 
scores are related to one or more criterion 
variables, information about the suitability 
and technical quality of the criteria should 
be reported. 

Comment: The description of each criterion 
variable should include evidence concerning 
its reliability, the extent to which it represents 
the intended construct (e.g., job performance), 
and the extent to which it is likely to be 
influenced by extraneous sources of variance. 
Special attention should be given to sources 
that previous research suggests may introduce 
extraneous variance that might bias the crite¬ 
rion for or against identifiable groups. 

Standard 1.17 

If test scores are used in conjunction with 
other quantifiable variables to predict some 
outcome or criterion, regression (or equiva¬ 
lent) analyses should include those additional 
relevant variables along with the test scores. 

Comment: In general, if several predictors of 
some criterion are available, the optimum 
combination of predictors cannot be deter¬ 
mined sotely from separate, pairwise examina¬ 
tions of the criterion variable with each 
separate predictor in turn. It is often informa¬ 
tive to estimate the increment in predictive 
accuracy that may be expected when each 
variable, including the test score, is intro¬ 
duced in addition to all other available vari¬ 
ables. Analyses involving multiple predictors 
should be verified by cross-validation or 
equivalent analysis whenever feasible, and the 
precision of estimated regression coefficients 
should be reported. 

Standard 1.18 

When statistical adjustments, such as those 
for restriction of range or attenuation, are 
made, both adjusted and unadjusted coeffi- 
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cients, as well as the specific procedure used, 
and all statistics used in the adjustment, 
should be reported. 

Comment: The correlation between two vari¬ 
ables, such as test scores and criterion meas¬ 
ures, depends on the range of values on each 
variable. For example, the test scores and the 
criterion values of selected applicants will typi¬ 
cally have a smaller range than the scores of 
all applicants. Statistical methods are available 
for adjusting the correlation to reflect the 
population of interest rather than the sample 
available. Such adjustments are often appro¬ 
priate, as when comparing results across 
various situations. Reporting an adjusted 
correlation should be accompanied by a state¬ 
ment of the method and the statistics used in 
making the adjustment. 

Standard 1.19 

If a test is recommended for use in assigning 
persons to alternative treatments or is likely 
to be so used, and if outcomes from those 
treatments can reasonably be compared on a 
common criterion, then, whenever feasible, 
supporting evidence of differential outcomes 
should be provided. 

Comment: If a test is used for classification 
into alternative occupational, therapeutic, or 
educational programs, it is not sufficient just 
to show that the test predicts treatment out¬ 
comes. Support for the validity of the classifi¬ 
cation procedure is provided by showing that 
the tesc is useful in determining which per¬ 
sons are likely ro profit differentially from 
one treatment or another. Treatment cate¬ 
gories may have to be combined to assemble 
sufficient cases for statistical analysis. It is rec¬ 
ognized, however, that such research may not 
be feasible, because ethical and legal con¬ 
straints on differential assignments may for¬ 
bid control groups. 


Standard 1.20 

When a meta-analysis is used as evidence of 
the strength of a test-criterion relationship, 
the test and the criterion variables in the 
local situation should be comparable with 
those in the studies summarized. If relevant 
research includes credible evidence that any 
other features of the testing application may 
influence the strength of the test-criterion 
relationship, the correspondence between 
those features in the local situation and in 
the meta-analysis should be reported. Any 
significant disparities that might limit the 
applicability of the meta-analytic findings to 
the local situation should be noted explicitly. 

Comment: The meta-analysis should incorpo¬ 
rate all available studies meeting explicitly 
stated inclusion criteria. Meta-analytic evi¬ 
dence used in test validation typically is based 
on a number of tests measuring the same or 
very similar constructs and criterion measures 
that likewise measure the same or similar 
constructs. A meta-analytic study may also be 
limited to a single test and a single criterion. 
For each study included in the analysis, the 
test-criterion relationship is expressed in some 
common metric, often as an effect size. The 
strength of the test-criterion relationship may 
be moderated by features of the situation in 
which the test and criterion measures were 
obtained (e.g., types of jobs, characteristics of 
test takers, time interval separating collection 
of test and criterion measures, year or decade 
in which the data were collected). If test-cri¬ 
terion relationships vary according to such 
moderator variables, then, the numbers of 
studies permitting, the meta-analysis should 
report separate estimated effect size distribu¬ 
tions conditional upon relevant situational 
features. This might be accomplished, for 
example, by reporting separate distributions 
for subsets of studies or by estimating the 
magnitudes of the influences of situational 
features on effect sizes. 
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Standard 1.21 

Any meta-analytic evidence used to support 
an intended test use should be clearly 
described, including methodological choices 
in identifying and coding studies, correcting 
for artifacts, and examining potential mod¬ 
erator variables. Assumptions made in cor¬ 
recting for artifacts such as criterion 
unreliability and range restriction should be 
presented, and the consequences of these 
assumptions made clear. 

Comment: Meta-analysis inevitably involves 
judgments regarding a number of method¬ 
ological choices. The bases for these judg¬ 
ments should be articulated. In the case of 
choices involving some degree of uncertainty, 
such as artifact corrections based on assumed 
values, the uncertainty should be acknowl¬ 
edged and the degree to which conclusions 
about validity hinge on these assumptions 
should be examined and reported. 

Standard 1.22 

When it is clearly stated or implied that a 
recommended test use will result in a specif¬ 
ic outcome, the basis for expecting that out¬ 
come should be presented, together with 
relevant evidence. 

Comment: If ir is asserted, for example, that 
using a given test for employee selection will 
result in reduced employee errors or training 
costs, evidence in support of that assertion 
should be provided. A given claim for the 
benefits of test use may be supported by logi¬ 
cal or theoretical argument as well as empiri¬ 
cal data. Due weight should be given to 
findings in the scientific literature that may 
be inconsistent with the stated expectation. 

Standard 1.23 

When a test use or score interpretation is 
recommended on the grounds that testing or 


the testing program per se will result in 
some indirect benefit in addition to the util¬ 
ity of information from the test scores them¬ 
selves, the rationale for anticipating the 
indirect benefit should be made explicit. 
Logical or theoretical arguments and empiri¬ 
cal evidence for the indirect benefit should 
be provided. Due weight should be given to 
any contradictory findings in the scientific 
literature, including findings suggesting 
important indirect outcomes other than 
those predicted. 

Comment: For example, certain educational 
testing programs have been advocated on 
the grounds that they would have a salutary 
influence on classroom instructional practices 
or would clarify students’ understanding of 
the kind or level of achievement they were 
expected to attain. To the extent that such 
claims enter into the justification for a testing 
program, they become part of the validity 
argument for test use and so should be exam¬ 
ined as part of the validation effort. Due 
weight should be given to evidence against 
such predictions, for example, evidence that 
under some conditions educational testing 
may have a negative effect on classroom 
instruction. 

Standard 1.24 

When unintended consequences result from 
test use, an attempt should be made to 
investigate whether such consequences arise 
from the test’s sensitivity to characteristics 
other than those it is intended to assess or 
to the test’s failure fully to represent the 
intended construct. 

Comment: The validity of test score interpre¬ 
tations may be limited by construct-irrelevant 
components or construct underrepresentation. 
When unintended consequences appear to 
stem, at least in part, from the use of one or 
more tests, it is especially important to check 
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that these consequences do not arise from 
such sources of invalidity. Although group 
differences, in and of themselves, do not call 
into question the validity of a proposed inter¬ 
pretation, they may increase the salience of 
plausible rival' hypotheses that should be 
investigated as part of the validation effort. 
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Background 

A test, broadly defined, is a set of tasks designed 
to elicit or a scale to describe examinee behavior 
in a specified domain, or a system for collecting 
samples of an individual’s work in a particular 
area. Coupled with the device is a scoring pro¬ 
cedure that enables the examiner to quantify, 
evaluate, and interpret the behavior or work 
samples. Reliability refers to the consistency 
of such measurements when the testing pro¬ 
cedure is repeated on a population of individ¬ 
uals or groups. 

The discussion that follows introduces 
concepts and procedures that may not be famil¬ 
iar to some readers. It is not expected that the 
brief definitions and explanations presented 
here will be sufficient to enable the less sophis¬ 
ticated reader to become adequately conver¬ 
sant with these developments. To achieve a 
better understanding, such readers may need 
to consult more comprehensive treatments 
in the measurement literature. 

The usefulness of behavioral measure¬ 
ments presupposes that individuals and groups 
exhibit some degree of stability in their behav¬ 
ior. However, successive samples of behavior 
from the same person are rarely identical in all 
pertinent respects. An individual’s perform¬ 
ances, products, and responses to sets of test 
questions vary in their quality or character 
from one occasion to another, even under 
strictly controlled conditions. This variation 
is reflected in the examinee’s scores. The caus¬ 
es of this variability are generally unrelated to 
the purposes of measurement. An examinee 
may try harder, may make luckier guesses, be 
more alert, feel less anxious, or enjoy better 
health on one occasion than another. An 
examinee may have knowledge, experience, or 
understanding that is more relevant to some 
tasks than to others in the domain sampled 
by the test. Some individuals may exhibit less 


variation in their scores than others, but no 
examinee is completely consistent. Because of 
•this variation and; in some instances, because 
of subjectivity in the scoring process, an indi¬ 
vidual’s obtained score and the average score 
of a group will always reflect at least a small 
amount of measurement error. 

To say that a score includes a component 
of error implies that there is a hypothetical 
error-free value that characterizes an examinee 
at the time of testing. In classical test theory 
this error-free value is referred to as the per¬ 
son’s true score for the test or measurement 
procedure. It is conceptualized as the hypo¬ 
thetical average score resulting from many 
repetitions of the test or alternate forms of 
the instrument. In statistical terms, the true 
score is a personal parameter and each observed 
score of an examinee is presumed to estimate 
this parameter. Under an approach to reliability 
estimation known as generalizability theory, a 
comparable concept is referred to as an exami¬ 
nee’s universe score. Under item response theory 
(IRT), a closely related concept is called an 
examinee’s ability or trait parameter, though 
observed scores and trait parameters may be 
stated in different units. The hypothetical dif¬ 
ference between an examinee’s observed score 
on any particular measurement and the exam¬ 
inee’s true or universe score for the procedure 
is called measurement error. 

The definition of what constitutes a 
standardized test or measurement procedure 
has broadened significantly in recent years. At 
one time the cardinal features of most stan¬ 
dardized tests were consistency of che test 
materials from examinee to examinee, close 
adherence to stipulated procedures for test 
administration, and use of prescribed scoring 
rules that could be applied with a high degree 
of consistency. These features were, in fact, 
what made a test “standardized,” and they 
made meaningful norms possible. In employ- 
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merit settings and certification programs, flex¬ 
ible measurement procedures have been in 
use for many years. Individualized oral exami¬ 
nations, simulations, analyses of extended 
case reports, and performance in real-life set¬ 
tings such as clinics are now commonplace. 

In education, however, large-scale testing pro¬ 
grams with a high degree of flexibility in test 
format and administrative procedures are a 
relatively recent development. In some pro¬ 
grams cumulative portfolios of student work 
have been substituted for more traditional 
end-of-year tests of achievement. Other pro¬ 
grams now allow examinees to choose their 
own topics to demonstrate their abilities. Still 
others permit or encourage small groups of 
examinees to work cooperatively in complet¬ 
ing the test. A science examination, for exam¬ 
ple, might involve a team of high school 
scudents who conduct a study of the sources 
of pollution in local streams and prepare a 
report on their findings. Examinations of 
this kind raise complex issues regarding the 
domain represented by the test and about 
the generalizability of individual and group 
scores. Each step toward greater flexibility 
almost inevitably enlarges the scope and mag¬ 
nitude of measurement error. However, it is 
possible that some of the resultant sacrifices 
in reliability may reduce construct irrelevance 
or construct underrepresentation in an assess¬ 
ment program. 

Characteristics and Implications of 
Measurement Error 

Errors of measurement are generally viewed as 
random and unpredictable. They are concep¬ 
tually distinguished from systematic errors, 
which may also affect performance of individ¬ 
uals or groups, but in a consistent rather than 
a random manner. For example, a systematic 
group error would occur as a result of differ¬ 
ences in the difficulty of test forms that have 
not been adequately equated. When one test 
form is less difficult than another, examinees 


who take the easier form may be expected to 
earn a higher average score than those who take 
the more difficult form. Such a difference 
would not be considered an error of measure¬ 
ment under most methods of quantifying and 
summarizing error, though generalizability 
theory would permit test form differences to 
be recognized as an error source. 

The systematic factors that may differen¬ 
tially affect the performance of individual test 
takers are not as easily detected or overridden 
as those affecting groups. For example, some 
examinees experience levels of test anxiety 
that severely impair cognitive efficiency. The 
presence of such a condition can sometimes 
be recognized in an examinee, but the effect 
cannot be overcome by statistical adjustments. 
The individual systematic errors are not gen¬ 
erally regarded as an element that contributes 
to unreliability. Rather, they constitute a 
source of construct-irrelevant variance and 
thus may detract from validity. 

Important sources of measurement error 
may be broadly categorized as chose rooted 
within the examinees and those external to 
them. Fluctuations in the level of an exam¬ 
inee’s motivation, interest, or attention and 
the inconsistent application of skills are clear¬ 
ly internal factors that may lead to score 
inconsistencies. Differences among testing 
sites in their freedom from distractions, the 
random effects of scorer subjectivity, and vari¬ 
ation in scorer standards are examples of 
external factors. The potency and importance 
of any particular source depend on the specif¬ 
ic conditions under which the measures are 
taken, how performances are scored, and the 
interpretadons made from the scores. A partic¬ 
ular factor, such as the subjectivity in scoring, 
may be a significant source of measurement 
error in some assessments and a minor con¬ 
sideration in others. 

Some changes in scores from one occa¬ 
sion to another, it should be noted, are not 
regarded as error, because they result, in part, 
from an intervention, learning, or maturation 
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that has occurred between the initial and final 
measures. The difference within an individual 
indicates, to some extent, the effects of the 
intervention or the extent of growth. In such 
settings, change per se constitutes the phe¬ 
nomenon of interest. The difference or the 
change score then becomes the measure to 
which reliability pertains. 

Measurement error reduces the useful¬ 
ness of measures. It limits the extent to which 
test results can be generalized beyond the par¬ 
ticulars of a specific application of the meas¬ 
urement process. Therefore, it reduces the 
confidence that can be placed in any single 
measurement. Because random measurement 
errors are inconsistent and unpredictable, 
they cannot be removed from observed 
scores. However, their aggregate magnitude 
can be summarized in several ways, as dis¬ 
cussed below. 

Summarizing Reliability Data 

Information about measurement error is 
essential to the proper evaluation and use of 
an instrument. This is true whether the meas¬ 
ure is based on the responses to a specific set 
of questions, a portfolio of work samples, the 
performance of a task, or the creation of an 
original product. The ideal approach to the 
study of reliability entails independent repli¬ 
cation of the entire measurement process. 
However, only a rough or partial approxima¬ 
tion of such replication is possible in many 
testing situations, and investigation of measure¬ 
ment error may require special studies that depart 
from routine testing procedures. Nevertheless, 
it should be the goal of test developers to 
investigate test reliability as fully as practical 
considerations permit. No test developer is 
exempt from this responsibility. 

The critical information on reliability 
includes the identification of the major 
sources of error, summary statistics bearing 
on the size of such errors, and the degree of 
generalizability of scores across alternate 


forms, scorers, administrations, or other rele¬ 
vant dimensions. It also includes a description 
of the examinee population to whom the 
foregoing data apply, as the data may accu¬ 
rately reflect what is true of one population 
but misrepresent what is true of another. For 
example, a given reliability coefficient or esti¬ 
mated standard error derived from scores of a 
nationally representative sample may differ 
significantly from that obtained for a more 
homogeneous sample drawn from one gen¬ 
der, one ethnic group, or one community. 

Reliability information may be reported 
in terms of variances or standard deviations of 
measurement errors, in terms of one or more 
coefficients, or in terms of IRT-based test 
information functions. The standard error of 
measurement is the standard deviation of a 
hypothetical distribution of measurement 
errors that arises when a given population is 
assessed via a particular test or procedure. 
The overall variance of measurement errors is 
actually a weighted average of the values that 
hold at various true score levels. The variance 
at a particular level is called a conditional 
error variance and its square root a conditional 
standard error. Traditionally, three broad cate¬ 
gories of reliability coefficients have been rec¬ 
ognized: (a) coefficients derived from the 
administration of parallel forms in independent 
testing sessions (alternate-form coefficients): 
(b) coefficients obtained by administration 
of the same instrument on separate occa¬ 
sions (tesc-recesc or stability coefficients); 
and (c) coefficients based on the relation¬ 
ships among scores derived from individual 
items or subsets of the items within a test, 
all data accruing from a single administra¬ 
tion (internal consistency coefficients). 
Where test scoring involves a high level of 
judgment, indexes of scorer consistency are 
commonly obtained. With the development 
of generalizability theory, the foregoing 
three categories may now be seen as special 
cases of a more general classification: gener¬ 
alizability coefficients. 
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Like traditional reliability coefficients, a 
generalizability coefficient is defined as the ratio 
of true or universe score variance to observed 
score variance. Unlike traditional approaches 
to the study of reliability, however, general iz- 
abiiity theory permits the researcher to specify 
and estimate the various components of true 
score variance, error variance, and observed 
score variance. Estimation is typically accom¬ 
plished by the application of the techniques 
of analysis of variance. Of special interest are 
the separate numerical estimates of the com¬ 
ponents of overall error variance. Such esti¬ 
mates permit examination of the contribution 
of each source of error to the overall measure¬ 
ment process. The generalizabilicy approach 
also makes possible the estimation of coeffi¬ 
cients that apply to a wide variety of potential 
measurement designs. 

The test information function, an impor¬ 
tant result of IRT, efficiently summarizes how 
well the test discriminates among individuals 
at various levels of the ability or trait being 
assessed. Under the IRT conceptualization, a 
mathematical function called the item charac¬ 
teristic curve or item response function is used 
as a model to represent the increasing propor¬ 
tion of correct responses to an item for groups 
at progressively higher levels of the ability or 
trait being measured. Given an adequate 
database, the parameters of the characteristic 
curve of each item in a test can be estimated. 
The test information function can then be 
approximated. This function may be viewed 
as a mathematical statement of the precision 
of measurement at each level of the given 
trait. Precision, in the IRT context, is analo¬ 
gous to the reciprocal of the conditional error 
variance of classical test theory. 

Interpretation of Reliability Data 

In general, reliability coefficients are most useful 
in comparing tests or measurement procedures, 
particularly those that yield scores in different 
units or metrics. However, such comparisons 


are rarely straightforward. Allowance must be 
made for differences in the variability of the 
groups on which the coefficients are based, 
the techniques used to obtain the coefficients, 
the sources of error reflected in the coeffi¬ 
cients; and theMengths of the instruments 
being compared in terms of testing time. 

Generalizability coefficients and the 
many coefficients included under the tradi¬ 
tional categories may appear to be inter¬ 
changeable, but some convey quite different 
information from others. A coefficient in any 
given category may encompass errors of 
measurement from a highly restricted per¬ 
spective, a very broad perspective, or some 
point between these extremes. For example, 
a coefficient may reflect error due to scorer 
inconsistencies but not reflect the variation 
that characterizes a succession of examinee 
performances or products. A coefficient may 
reflect only the internal consistency of item 
responses within an instrument and fail to 
reflect measurement error associated with 
day-to-day changes in examinee health, effi¬ 
ciency, or motivation. 

It should not be inferred, however, that 
alternate-form or test-retest coefficients based 
on test administrations several days or weeks 
apart are always preferable to internal consis¬ 
tency coefficients. For many tests, internal 
consistency coefficients do not differ signifi¬ 
cantly from alternate-form coefficients. Where 
only one form of a test exists, retesting may 
result in an inflated correlation between the 
first and second scores due to idiosyncratic 
features of the test or to examinee recall of 
initial responses. Also, an individual’s status 
on some attributes, such as mood or emo¬ 
tional state, may change significantly in a 
short period of time. In the assessment of 
such constructs the multiple measures that 
give rise to reliability estimates should be 
obtained within the short period in which the 
attribute remains stable. Therefore, for char¬ 
acteristics of this kind an internal consistency 
coefficient may be preferred. 
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The standard error of measurement is 
generally more relevant than the reliability 
coefficient once a measurement procedure has 
been adopted and interpretation of scores has 
become the user’s primary concern. It should 
be noted that standard errors share some of 
the ambiguities which characterize reliability 
coefficients, and estimates may vary in their 
quality. Information about the precision of 
measurement at each of several widely spaced 
score levels—-that is, conditional standard 
errors—is usually a valuable supplement to the 
single statistic for all score levels combined. 
Like reliability and generalizability coeffi¬ 
cients, standard errors may reflect variation 
from many sources of error or only a few. 
For most purposes, a more comprehensive 
standard error is more informative than a 
less comprehensive value. However, there 
are many exceptions to this generalization. 
Practical constraints often preclude conduct 
of the kinds of studies that would yield esti¬ 
mates of the preferred standard errors. 

Measurements derived from observations 
of behavior or evaluations of products are espe¬ 
cially sensitive to a variety of error factors. These 
include evaluator biases and idiosyncrasies, 
scoring subjectivity, and intra-examinee factors 
that cause variation from one performance or 
product to another. The methods of general¬ 
izability theory are well suited to the investi¬ 
gation of the reliability of the scores on such 
measures. Estimates of the error variance 
associated with each specific source and with 
the interactions between sources indicate the 
extent to which examinee scores may be gen¬ 
eralized to a population of scorers and ro a 
universe of products or performances. 

The interpretations of test scores may be 
broadly categorized as relative or absolute. 
Relative interpretations convey the standing 
of an individual or group within a reference 
population. Absolute interpretations relate the 
status of an individual or group to defined 
standards. These standards may originate in 
empirical data for one or more populations or 


be based entirely on authoritative judgment. 
Different values of the standard error apply 
to the two types of interpretations. 

The test information function can be 
perceived an alternative to traditional indices 
of measurement precision, but there are 
important distinctions that should be noted. 
Standard errors under classical test theory can 
be derived by several different approaches. 
These yield similar, but not identical, results. 
More significantly, standard errors, like relia¬ 
bility coefficients, may reflect a broad con¬ 
figuration of error factors or a restricted 
configuration, depending on the design of the 
reliability study. Test information functions, 
on the other hand, are limited to the restrict¬ 
ed definition of measurement error that is 
associated with internal consistency reliabili¬ 
ties. In addition, under IRT several different 
mathematical models have been proposed and 
accepted as the basic form of the item charac¬ 
teristic curve. Adoption of one model rather 
than another can have a material effect on the 
derived test information function. 

A final consideration has significant impli¬ 
cations for both IRT and classical approaches 
to quantification of test score precision. It is 
this: Indices of precision depend on the scale 
in which they are reported. An index stated 
in terms of raw scores or the trait level esti¬ 
mates of IRT may convey a radically different 
perception of reliability than the same index 
restated in terms of derived scores. This same 
contrast may hold for conditional standard 
errors. In terms of the basic score scale, preci¬ 
sion may appear to be high at one score level, 
low at another. But when the conditional 
standard errors are restated in units of derived 
scores, such as grade equivalents or standard 
scores, quite different trends in comparative 
precision may emerge. Therefore, measure¬ 
ment precision under both theories very 
strongly depends on the scale in which test 
scores are reported and interpreted. 

Precision and consistency in measure¬ 
ment are always desirable. However, the need 
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for precision increases as the consequences of 
decisions and interpretations grow in impor¬ 
tance. If a decision can and will be corrobo¬ 
rated by information from other sources or if 
an erroneous initial decision can be quickly 
corrected, scores with modest reliability may 
suffice. But if a test score leads to a decision 
that is not easily reversed, such as rejection or 
admission of a candidate to a professional 
school or the decision by a jury that a serious 
injury was sustained, the need for a high degree 
of precision is much greater. 

Where the purpose of measurement is 
classification, some measurement errors are 
more serious than others. An individual who 
is far above or far below the value established 
for pass/fail or for eligibility for a special pro¬ 
gram can be mismeasured without serious 
consequences. Mismeasurement of examinees 
whose true scores are close to the cut score is 
a more serious concern. The techniques used 
to quantify reliability should recognize these 
circumstances. This can be done by reporting 
the conditional standard error in the vicinity 
of the critical value. 

Some authorities have proposed that a 
semantic distinction be made between “relia¬ 
bility of scores” and “degree of agreement in 
classification." The former term would be 
reserved for analysis of score variation under 
repeated measurement. The term classification 
consistency or inter-rater agreement, rather than 
reliability, would be used in discussions of 
consistency of classification. Adoption of such 
usage would make it dear that the impor¬ 
tance of an error of any given size depends on 
the proximity of the examinees score to the 
cut score. However, it should be recognized 
that the degree of consistency or agreement in 
examinee classification is specific to the cut 
score employed and its location within the 
score distribution. 

Average scores of groups, when interpret¬ 
ed as measures of program effectiveness, 
involve error factors that are not identical to 
those that operate at the individual level. For 


large groups, the positive and negative meas¬ 
urement errors of individuals may average out 
almost completely in group means. However, 
the sampling errors associated with the ran¬ 
dom sampling of persons who are tested for 
purposes of program evaluation are still pres¬ 
ent. This component of the variation i n the 
mean achievement of school classes from year 
to year or in the average expressed satisfaction 
of successive samples of the clients of a pro¬ 
gram may constitute a potent source of error 
in program evaluations. It can be a significant 
source of error in inferences about programs 
even if there is a high degree of precision in 
individual test scores. Therefore, when an 
instrument is used to make group judgments, 
reliability data must bear directly on the 
interpretations specific to groups. Standard 
errors appropriate to individual scores are not 
appropriate measures of the precision of group 
averages. A more appropriate statistic is the 
standard error of the observed score means. 
Generalizability theory can provide more 
refined indices when the sources of measure¬ 
ment error are numerous and complex. 

Typically, developers and distributors of 
tests have primary responsibility for obtain¬ 
ing and reporting evidence of reliability or 
test information functions. The user must 
have such data to make an informed choice 
among alternative measurement approaches 
and will generally be unable to conduct relia¬ 
bility studies prior to operational use of an 
instrument. In some instances, however, local 
users of a test or procedure must accept at 
least partial responsibility for documenting 
the precision of measurement. This obliga¬ 
tion holds when one of the primary purposes 
of measurement is to rank or classify exam¬ 
inees within the local population. It also 
holds when users must rely on local scorers 
who are trained to use the scoring rubrics 
provided by the test developer. In such set¬ 
tings, local factors may materially affect the 
magnitude of error variance and observed 
score variance. Therefore, the reliability of 
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scores may differ appreciably from that report¬ 
ed by the developer. 

The reporting of reliability coefficients 
alone, with little detail regarding the methods 
used to estimate the coefficient, the narure of 
the group from which the data were derived, 
and the conditions under which the data were 
obtained constitutes inadequate documentation. 
General statements to the effect that a test is 
"reliable" or that it is “sufficiently reliable to 
permit interpretations of individual scores” are 
rarely, if ever, acceptable. It is the user who must 
take responsibility for determining whether or 
not scores are sufficiently trustworthy to justify 
anticipated uses and interpretations. Of course, 
test constructors and publishers are obligated 
to provide sufficient data to make informed 
judgments possible. 

As the foregoing comments emphasize, 
there is no single, preferred approach to 
quantification of reliability. No single index 
adequately conveys all of the relevant facts. 

No one method of investigation is optimal in 
all situations, nor is the test developer limited 
to a single approach for any instrument. The 
choice of estimation techniques and the mini¬ 
mum acceptable level for any index remain a 
matter of professional judgment. 

Although reliability is discussed here as an 
independent characteristic of test scores, it should 
be recognized that the level of reliability of scores 
has implications for the validity of score inter¬ 
pretations. Reliability data ultimately bear on 
the repeatability of the behavior elicited by the 
test and the consistency of the Fesultant scores. 
The data also bear on the consistency of classi¬ 
fications of individuals derived from the scores. 
To the extent that scores reflect random errors 
of measurement, their potential for accurate 
prediction of criteria, for beneficial examinee 
diagnosis, and for wise decision making is lim¬ 
ited. Relatively unreliable scores, in conjunction 
with other convergent information, may some¬ 
times be of value to a test user, but the level of 
a score’s reliability places limits on its unique 
contribution to validity for all purposes. 


Standard 2.1 

For each total score, subscore, or combina¬ 
tion of scores that is to be interpreted, esti¬ 
mates of relevant reliabilities and standard 
errors of measurement or test information 
functions should be reported. 

Comment: It is not sufficient to report esti¬ 
mates of reliabilities and standard errors of 
measurement only for total scores when sub- 
scores are also interpreted. The form-to-form 
and day-to-day consistency of total scores on 
a test may be acceptably high, yet subscores 
may have unacceptably low reliability. For all 
scores to be interpreted, users should be sup¬ 
plied with reliability data in enough detail to 
judge whether scores are precise enough for 
the users’ intended interpretations. Composites 
formed from selected subtests within a test 
battery are frequently proposed for predictive 
and diagnostic purposes. Users need informa¬ 
tion about the reliability of such composites. 

Standard 2.2 

The standard error of measurement, both 
overall and conditional (if relevant), should 
be reported both in raw score or original 
scale units and in units of each derived score 
recommended for use in test interpretation. 

Comment: The most common derived scores 
include standard scores, grade or age equiva¬ 
lents, and percentile tanks. Because raw scores 
on norm-referenced tests ate only rarely inter¬ 
preted directly, standard errors in derived 
score units are more helpful to the typical test 
user. A confidence interval for an examinee's 
true score, universe score, or percentile rank 
serves much the same purpose as a standard 
error and can be used as an alternative approach 
to convey reliability information. The impli¬ 
cations of the standard error of measurement 
are especially important in situations where 
decisions cannot be postponed and corrobo¬ 
rative sources of information are limited. 
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Standard 2.3 

When test interpretation emphasizes differ¬ 
ences between two observed scores of an 
individual or two averages of a group, relia¬ 
bility data, including standard errors, should 
be provided for such differences. 

Comment: Observed score differences are used 
for a variety of purposes. Achievement gains 
are frequently the subject of inferences for 
groups as well as individuals. Differences 
between verbal and performance scores of 
intelligence and scholastic ability tests are 
often employed in the diagnosis of cognitive 
impairment and learning problems. Psycho¬ 
diagnostic inferences are frequently drawn 
from the differences between subtest scores. 
Aptitude and achievement batteries, interest 
inventories, and personality assessments are 
commonly used to identify and quantify the 
relative strengths and weaknesses or the pat¬ 
tern of trait levels of an examinee. When the 
interpretation of test scores centers on the 
peaks and valleys in the examinees test score 
profile, the reliability of score differences for 
all pairs of scores is critical. 

Standard 2.4 

Each method of quantifying the precision 
or consistency of scores should be described 
clearly and expressed in terms of statistics 
appropriate to the method. The sampling 
procedures used to select examinees for relia¬ 
bility analyses and descriptive statistics on 
these samples should be reported. 

Comment: Information on the method of 
subject selection, sample sizes, means, stan¬ 
dard deviations, and demographic characteris¬ 
tics of the groups helps users judge the extent 
to which reported data apply to their own 
examinee populations. If the test-retest or 
alternate-form approach is used, the interval 
between testings should be indicated. Because 
there are many ways of estimating reliability, 

32 


each influenced by different sources of meas¬ 
urement error, it is unacceptable to say simply, 
“The reliability of test X is .90." A better 
statement would be, “The reliability coeffi¬ 
cient of .90 reported for scores on test X was 
obtained by correlating scores from forms A 
and B administered on successive days. The 
data were based on a sample of 400 lOth-grade 
students from five middle-class suburban 
schools in New York State. The demographic 
breakdown of this group was as follows: ...." 

Standard 2.5 

A reliability coefficient or standard error of 
measurement based on one approach should 
not be interpreted as interchangeable with 
another derived by a different technique 
unless their implicit definitions of measure¬ 
ment error are equivalent. 

Comment: Internal consistency, alternate- 
form, test-retest, and generaiizability coeffi¬ 
cients should not be considered equivalent, as 
each may incorporate a unique definition of 
measurement error. Error variances derived 
via item response theory may not be equiva¬ 
lent to error variances estimated via other 
approaches. Test developers should indicate 
the sources.of error that are reflected in or 
ignored by the reported reliability indices. 

Standard 2.6 

If reliability coefficients are adjusted for restric¬ 
tion of range or variability, the adjustment pro¬ 
cedure and both the adjusted and unadjusted 
coefficients should be reported. The standard 
deviations of the group actually tested and of 
the target population, as well as the rationale 
for the adjustment, should be presented. 

Comment: Application of a correction for 
restriction in variability presumes that the 
available sample is not representative of the 
cest-taker population to which users might be 
expected to generalize. The rationale for the 
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correction should consider the appropriate¬ 
ness of such a generalization. Adjustment for¬ 
mulas that presume constancy in the standard 
error across score levels should not be used 
unless constancy can be defended. 

Standard 2.7 

When subsets of items within a test are dic¬ 
tated by the test specifications and can be 
presumed to measure partially independent 
traits or abilities, reliability estimation pro¬ 
cedures should recognize the multifactor 
character of the instrument. 

Comment: The total score on a test that is 
dearly multifactor in nature should be treated 
as a composite score. If an internal consistency 
estimate of total score reliability is obtained 
by the split-halves procedure, the halves 
should be parallel in content and statistical 
characteristics. Stratified coefficient alpha 
should be used rather than the more familiar 
nonstratified coefficient. 

Standard 2.8 

Test users should be informed about the 
degree to which rate of work may affect 
examinee performance. 

Comment: It is not possible to state, in general, 
whether reliability coefficients will increase or 
decrease when rate of work becomes an impor¬ 
tant source of systematic variance. Rate of work, 
as an examinee trait, may be more stable or 
less stable from occasion to occasion than the 
other factors the test is designed to measure. 
Because speededness has differential effects on 
various estimates, information on speededness 
is helpful in interpreting reported coefficients. 

The importance of the speed factor can 
sometimes be inferred from analyses of item 
responses and from observations by examiners 
during test administrations conducted for 
reliability analyses. The distribution of “last 
item attempted” and increases in the frequen¬ 


cy of omitted responses toward the end of a 
test are also highly informative, though not 
conclusive, evidence regarding speededness. A 
decline in the proportion of correct responses, 
beyond that attributable to increasing item 
difficulty, may indicate that some examinees 
were responding randomly. With computer- 
administered tests, abnormally fast item response 
times, particularly toward the end of the test, 
may also suggest that examinees were respond¬ 
ing randomly. In the case of constructed- 
response exercises, including essay questions, 
the completeness of the responses may sug¬ 
gest that time constraints had little effect on 
early items but a significant effect on lacer 
items. Introduction of a speed factor inco 
what might otherwise be a power test may 
have a marked effect on alternate-form and 
test-retest reliabilities. A shift from a paper- 
and-pencil format to a computer-adminis¬ 
tered format may affect test speededness. 

Standard 2.9 

When a test is designed to reflect rate of 
work, reliability should be estimated by the 
alternate-form or test-retest approach, using 
separately timed administrations. 

Comment: Split-half coefficients based on 
separate scores from the odd-numbered and 
even-numbered items are known to yield 
inflated estimates of reliability for highly 
speeded tests. Coefficient alpha and other 
internal consistency coefficients may also be 
biased, though the size of the bias is not as 
clear as that for the split-halves coefficient. 

Standard 2.10 

When subjective judgment enters into test 
scoring, evidence should be provided on both 
inter-rater consistency in scoring and within- 
examinee consistency over repeated measure¬ 
ments. A clear distinction should be made 
among reliability data based on (a) independ¬ 
ent panels of raters scoring the same perform- 
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ances or products, (b) a single panel scoring 
successive performances or new products, and 
(c) independent panels scoring successive per¬ 
formances or new products. 

Comment: Task-to-rask variations in the quality 
of an examinee's performance and rarer-to-rarer 
inconsistencies in scoring represent independ¬ 
ent sources of measurement error. Reports of 
reliability studies should make clear which of 
these sources are reflected in the data. Where 
feasible, the error variances arising from each 
source should be estimated. Generalizability 
studies and variance component analyses are 
especially helpful in chis regard. These analy¬ 
ses can provide separate error variance esti¬ 
mates for tasks within examinees, for judges, 
and for occasions within the time period of 
trait stability. Information should be provided 
on the qualifications of the judges used in 
reliability studies. 

Inter-rater or inter-observer agreement 
may be particularly important for ratings and 
observational data that involve subtle discrimi¬ 
nations. It should be noted, however, that 
when raters evaluate positively correlated 
characteristics, a favorable or unfavorable 
assessment of one trait may color their opin¬ 
ions of other traits. Moreover, high inter-rater 
consistency does not imply high examinee 
consistency from cask to cask. Therefore, 
internal consistency within raters and inter¬ 
rater agreement do not guarantee high relia¬ 
bility of examinee scores. 

Standard 2.11 

If there are generally accepted theoretical or 
empirical reasons for expecting that reliabili¬ 
ty coefficients, standard errors of measure¬ 
ment, or test information functions will 
differ substantially for various subpopula¬ 
tions, publishers should provide reliability 
data as soon as feasible for each major popu¬ 
lation for which the test is recommended. 


Comment: IF test score interpretation involves 
inferences within subpopulations as well as 
within the general population, reliability data 
should be provided for both the subpopulations 
and the general population. Test users who 
work exclusively with a specific cultural group 
or with individuals who have a particular dis¬ 
ability would benefit from an estimate of the 
standard error for such a subpopulation. Some 
groups of test takers—pre-school children, for 
example—tend to respond to test stimuli in a 
less consistent fashion than do older children. 

Standard 2.12 

If a test is proposed for use in several grades 
or over a range of chronological age groups 
and if separate norms are provided for each 
grade or each age group, reliability data should 
be provided for each age or grade population, 
not solely for all grades or ages combined. 

Comment: A reliability coefficient based on a 
sample of examinees spanning several grades 
or a broad range of ages in which average 
scores are steadily increasing will generally 
give a spuriously inflated impression of relia¬ 
bility. When a test is intended to discriminate 
within age or grade populations, reliability 
coefficients and standard errors should be 
reported separately for each population. 

Standard 2.13 

If local scorers are employed to apply gener¬ 
al scoring rules and principles specified by 
the test developer, local reliability data should 
be gathered and reported by local authorities 
when adequate size samples are available. 

Comment: For example, many statewide test¬ 
ing programs depend on local scoring of 
essays, constructed-response exercises, and 
performance tests. Reliability analyses bear on 
the possibility that additional training of scor¬ 
ers is needed and, hence, should be an inte¬ 
gral part of program monitoring. 
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Standard 2.14 

Conditional standard errors of measurement 
should be reported at several score levels if 
constancy cannot be assumed. Where cut scores 
are specified for selection or classification, the 
standard errors of measurement should be 
reported in the vicinity of each cut score. 

Comment: Estimation of conditional standard 
errors is usually feasible even with the sample 
sizes that are typically used for reliability 
analyses. If it is assumed that the standard 
error is constant over a broad range of score 
levels, the rationale for this assumption should 
be presented. 

Standard 2.15 

When a test or combination of measures is 
used to make categorical decisions, estimates 
should be provided of the percentage of 
examinees who would be classified in the 
same way on two applications of the proce¬ 
dure, using the same form or alternate forms 
of the instrument. 

Comment: When a test or composite is used to 
make categorical decisions, such as pass/fail, 
the standard error of measurement at or near 
the cut score has important implications for the 
trustworthiness of these decisions. However, 
the standard error cannot be translated into 
the expected percentage of consistent deci¬ 
sions unless assumptions are made about the 
form of the distributions of measurement 
errors and true scores. It is preferable that this 
percentage be estimated directly through the 
use of a repeated-measurements approach if 
consistent with the requirements of test secu¬ 
rity and if adequate samples are available. 

Standard 2.16 

In some testing situations, the items vary from 
examinee to examinee—through random selec¬ 
tion from an extensive item pool or application 


of algorithms based on the examinees level of 
performance on previous items or preferences 
with respect to item difficulty. In this type of 
testing, the preferred approach to reliability 
estimation is one based on successive adminis¬ 
trations of the test under conditions similar to 
those prevailing in operational test use. 

Comment: Varying the set of items presented 
to each examinee is an acceptable procedure 
in some settings. If this approach is used, reli¬ 
ability data should be appropriate to this pro¬ 
cedure. Estimates of standard errors of ability 
scores can be computed through the use of 
IRT and reported routinely as part of the 
adaptive testing procedure. However, those 
estimates are not an adequate substitute for 
estimates based on successive administrations 
of the adaptive test, nor do they bear on the 
issue of stability over short intervals. IRT esti¬ 
mates are contingent on the adequacy of both 
the item parameter estimates and the item res¬ 
ponse models adopted in the theory. Estimates 
of reliabilities and standard errors of measure¬ 
ment based on the administration and analysis 
of alternate forms of an adaptive test reflect 
errors associated with the entire measurement 
process. The alternate-form estimates provide 
an independent check on the magnitude of 
the errors of measurement specific to the 
adaptive feature of the testing procedure. 

Standard 2.17 

When a test is available in both long and short 
versions, reliability data should be reported for 
scores on each version, preferably based on an 
independent administration of each. 

Comment: Some tests and test batteries are 
published in both a “full-length” version and 
a “survey” or “short” version. In many appli¬ 
cations the Spearman-Brown formula will sat¬ 
isfactorily approximate the reliability of one of 
these from data based on the other. However, 
context effects are commonplace in tests of 
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maximum performance. Also, the short ver¬ 
sion of a standardized test often comprises a 
nonrandom sample of items from the full- 
length version. Therefore, the shorter version 
may be more reliable or less reliable than the 
Spearman-Brown projections from the full- 
length version: The reliability of scores on 
each version is best evaluated through an 
independent administration of each, using 
the designated time limits. 

Standard 2.18 

When significant variations are permitted in 
test administration procedures, separate reli¬ 
ability analyses should be provided for scores 
produced under each major variation if ade¬ 
quate sample sizes are available. 

Comment: To accommodate examinees with 
disabilities, test publishers might authorize 
modifications in the procedures and time 
limits that are specified for the administration 
of the paper-and-pencil edition of a test. In 
some cases, modified editions of the test itself 
may be provided. For example, tape-recorded 
versions for use in a group setting or with 
individual equipment may be used to test 
examinees who exhibit reading disabilities or 
attention deficits. If such modifications can 
be employed with test takers who are not dis¬ 
abled, insights can be gained regarding the 
possible effects on test scores of these non¬ 
standard administrations. 

Standard 2.19 

When average test scores for groups are used 
in program evaluations, the groups tested 
should generally be regarded as a sample 
from a larger population, even if all exam¬ 
inees available at the time of measurement are 
tested. In such cases the standard error of the 
group mean should be reported, as it reflects 
variability due to sampling of examinees as 
well as variability due to measurement error. 


Comment: The graduating seniors of a liberal 
arts college, the current clients of a social 
service agency, and analogous groups exposed 
to a program of interest typically constitute a 
sample in a longitudinal sense. Presumably, 
comparable groups from the same population 
will recur in future years, given static condi¬ 
tions. The factors leading to uncertainty in 
conclusions about program effectiveness arise 
from the sampling of persons as well as meas¬ 
urement error. Therefore, the standard error 
of the mean observed score, reflecting varia¬ 
tion in both true scores and measurement 
errors, represents a more realistic standard 
error in this setting. Even this value may 
underestimate the variability of group means 
over time. In many settings, the static condi¬ 
tions assumed under random sampling of 
persons do not prevail. 

Standard 2.20 

When the purpose of testing is to measure the 
performance of groups rather than individuals, 
a procedure frequently used is to assign a small 
subset of items to each of many subsamples of 
examinees. Data are aggregated across sub¬ 
samples and item subsets to obtain a measure 
of group performance. When such procedures 
are used for program evaluation or population 
descriptions, reliability analyses must take the 
sampling scheme into account. 

Comment: This type of measurement program 
is termed matrix sampling. It is designed to 
reduce the time demanded of individual 
examinees and to increase the total number of 
items on which data are obtained. This test¬ 
ing approach provides the same type of infor¬ 
mation about group performances that would 
accrue if all examinees could respond to all 
exercises in the item pool. Reliability statistics 
must be appropriate to the sampling plan 
used with respect to examinees and items. 
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Background 

Test development is the process of producing 
a measure of some aspect of an individual’s 
knowledge, skill, ability, interests, attitudes, 
or other characteristics by developing items 
and combining them to form a test, accord¬ 
ing to a specified plan. Test development is 
guided by the stated purpose(s) of the test 
and the intended inferences to be made from 
the test scores. The test development process 
involves consideration of content, format, the 
context in which the test will be used, and 
the potential consequences of using the test. 
Test development also includes specifying 
conditions for administering the test, deter¬ 
mining procedures for scoring the test per¬ 
formance, and reporting the scores to test 
takers and test users. This chapter focuses pri¬ 
marily on the following aspects of test devel¬ 
opment: stating the purposed) of the test, 
defining a framework for the test, developing 
test specifications, developing and evaluating 
items and their associated scoring procedures, 
assembling the test, and revising the test. The 
first section describes the test development 
process that begins with a statement of the 
purpose(s) of the test and culminates with 
the assembly of the test. The second section 
addresses several special considerations in test 
development, including considerations in 
delineating the test framework and in devel¬ 
oping performance assessments. The chapter 
concludes with a discussion on test revision. 
Issues bearing on validity, reliability, and fair¬ 
ness are interwoven within the stages of test 
development. Each of these topics is addressed 
comprehensively in other chapters of the 
Standards-, validity in chapter 1, reliability in 
chapter 2, and aspects of fairness in chapters 
7, 8, 9, and 10. Additional material on test 
administration and scoring, and on reporting 
scores and results, is provided in chapter 5. 
Chapter 4 discusses score scales, and the focus 
of chapter 6 is test documents. 


Test Development 

The process of developing educational and psy¬ 
chological tests commonly begins with a state¬ 
ment oft of the test arid the 

construct or content domain to be measured. 
Tests of the same construct or domain can dif¬ 
fer in important ways, because a number of 
decisions must be made as the test is developed. 
It is helpful to consider the four phases leading 
from the original statement of purpose(s) to the 
final product: (a) delineation of the purpose(s) 
of the test and the scope of the construct or the 
extent of the domain to be measured; (b) devel¬ 
opment and evaluation of the test specifica¬ 
tions; (c) development, field testing, evaluation, 
and selection of the items and scoring guides 
and procedures; and (d) assembly and evalua¬ 
tion of the test for operational use. What fol¬ 
lows is a description of typical test development 
procedures, though there may be sound reasons 
that some of these steps are followed in some 
settings and not in others. 

The first step is to extend the original 
statement of purpose(s), and the construct or 
content domain being considered, into a frame¬ 
work for the test that describes the extent of 
the domain, or the scope of the construct to 
be measured. The test framework, therefore, 
delineates the aspects (e.g., content, skills, 
processes, and diagnostic features) of the con¬ 
struct or domain to be measured. For example, 
“Does eighth-grade mathematics include 
algebra?” “Does verbal ability include text 
comprehension as well as vocabulary?” “Does 
self-esteem include both feelings and acts?” 
The delineation of the test framework can be 
guided by theory or an analysis of the content 
domain or job requirements as in the case of 
many licensing and employment tests. The test 
framework serves as a guide to subsequent test 
evaluation. The chapter on validity provides a 
more thorough discussion of the relationships 
among the construct or content domain, the 
test framework, and the purposed) of the test. 
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Once decisions have been made about 
what the test is to measure, and what its scores 
are intended to convey, the next step is to 
design the test by establishing test specifica¬ 
tions. The test specifications delineate the for¬ 
mat of items, tasks, or questions; the response 
format or conditions for responding; and the 
type of scoring procedures. The specifications 
may indicate the desired psychometric prop¬ 
erties of items, such as difficulty and discrimi¬ 
nation, as well as the desired test properties 
such as test difficulty, inter-item correlations, 
and reliability. The test specifications may 
also include such factors as time restrictions, 
characteristics of the intended population of 
test takers, and procedures for administration. 
All subsequent test development activities are 
guided by the test specifications. 

Test specifications will include, at least 
implicitly, an indication of whether the test 
scores will be primarily norm-referenced or 
criterion-referenced. When scores are norm- 
referenced, relative score interpretations are of 
primary interest. A score for an individual or 
for a definable group is ranked within one or 
more distributions of scores or compared to 
the average performance of test takers for var¬ 
ious reference populations (e.g., based on age, 
grade, diagnostic category, or job classifica¬ 
tion). When scores are criterion-referenced, 
absolute score interpretations are of primary 
interest. The meaning of such scores does not 
depend on rank information. Rather, the test 
score conveys directly a level of competence. 
in some defined criterion domain. Both rela¬ 
tive and absolute interpretations are often 
used with a given test, but the test developer 
determines which approach is most relevant 

The nature of the item and response for¬ 
mats that may be specified depends on the 
purposes of the test and the defined domain 
of the test. Selected-response formats, such as 
multiple-choice items, are suitable for many 
purposes of testing. The test specifications 
indicate how many alternatives are to be used 


for each item. Other purposes may be more 
effectively served by a short construcced-response 
format. Short-answer items require a response 
of no mote than a few words. Extended-response 
formats require the test taker to write a more 
extensive response of one or more sentences 
or paragraphs. Performance assessments ofren 
seek to emulate the context or conditions in 
which the intended knowledge or skills are 
actually applied. One type of performance 
assessment, for example, is the standardized 
job or work sample. A task is presented to the 
test taker in a standardized format under 
standardized conditions. Job or work samples 
might include, for example, the assessment of 
a practitioners ability to make an accurate diag¬ 
nosis and recommend treatment for a defined 
condition, a managers ability to articulate goals 
for an organization, or a students proficiency 
in performing a science laboratory experiment. 

All types of items require some indica¬ 
tion of how to score the responses. For select¬ 
ed-response items, one alternative is considered 
the correct response in some testing programs. 
In other testing programs, the alternatives may 
be weighted differentially. For short-answer 
items, a list of acceptable alternatives may 
suffice; extended-response items need more 
detailed rules for scoring, sometimes called 
scoring rubrics. Scoring rubrics specify the crite¬ 
ria for evaluating performance and may vary in 
the degree of judgment encailed, in the number 
of score levels, and in other ways. It is com¬ 
mon practice for test developers to provide 
scorers with examples of performances at each 
of the score levels to help clarify the criteria. 

For extended-response items, including 
performance tasks, cwo major types of scoring 
procedures are used: analytic and holistic. Both 
of the procedures require explicit performance 
criteria that reflea the test framework. However, 
the approaches differ in the degree of detail 
provided in the evaluation report. Under the 
analytic scoring procedure, each critical 
dimension of the performance criteria is judged 
independendy, and separate scores are obtained 
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for each of these dimensions in addition to 
an overall score. Under the holistic scoring 
procedure, the same performance criteria may 
implicitly be considered, but only one overall 
score is provided. Because the analytic proce¬ 
dure provides information on a number of 
critical dimensions, it potentially provides valu¬ 
able information for diagnostic purposes and 
lends itself to evaluating strengths and weak¬ 
nesses of test takers. In contrast, the holistic 
procedure may be preferable when an overall 
judgment is desired and when the skills being 
assessed are complex and highly interrelated. 
Regardless of the type of scoring procedure, 
designing the items and developing the scoring 
rubrics and procedures is an integrated process. 

A participatory approach may be used in 
the design of items, scoring rubrics, and some¬ 
times the scoring process itself. Many interested 
persons (e.g., practitioners, teachers) may be 
involved in developing items and scoring rubrics, 
and/or evaluating the subsequent performan¬ 
ces. If a participatory approach is used, partici¬ 
pants’ knowledge about the domain being 
assessed and their ability to apply the scoring 
rubrics are of critical importance. Equally 
important, for those involved in developing 
tests and evaluating performances, is their 
familiarity with the nature of the population 
being tested. Relevant characteristics of the 
population being tested may include the typi¬ 
cal range of expected skill levels, their famil¬ 
iarity with the response modes required of 
them, and the primary language they use. 

The test developer usually assembles an 
item pool that consists of a larger set of items 
than what is required by the test specifications. 
This allows for the test developer to select 
a set of items for the test that meet the test 
specifications. The quality of the items is 
usually ascertained through item review pro¬ 
cedures and pilot testing. Items are reviewed 
for content quality, clarity and lack of ambi¬ 
guity. Items sometimes are reviewed for sensi¬ 
tivity to gender or cultural issues. An attempt 
is generally made to avoid words and topics 


that may offend or otherwise disturb some 
test takers, if less offensive material is equally 
useful. Often, a field test is developed and 
administered to a group of test takers who are 
somewhat representative of the target popula¬ 
tion for the test. The field test helps deter¬ 
mine some of the psychometric properties of 
the test items, such as an items difficulty and 
ability to discriminate among test takers of 
different standing on the scale. Ongoing test¬ 
ing programs often pretest items by inserting 
them into existing tests. Those items are not 
used in obtaining test scores of the test takers, 
but the item responses provide useful data for 
test development. 

The next step in test development is to 
assemble items into a test or to identify an 
item pool for an adaptive test. The test devel¬ 
oper is responsible for ensuring that the items 
selected for the test meet the requirements of 
the test specifications. Depending upon the 
purpose(s) of the test, relevant considerations 
in item selection may include the content 
quality and scope, the weighting of items and 
subdomains, and the appropriateness of the 
items selected for the intended population of 
test takers. Often test developers will specify 
the distribution of psychometric indices of 
the items to be included in the test. For 
example, the specified distribution of item 
difficulty indices for a selection test would 
differ from the distribution specified for a 
general achievement test. When psychometric 
indices of the items are estimated using item 
response theory (IRT), the fit of the model 
to the data is also evaluated. This is accom¬ 
plished by evaluating the extent to which the 
assumptions underlying the item response 
model (e.g., unidimensionality, local inde¬ 
pendence, speededness, and equality of slope 
parameters) are satisfied. 

The test developer is also responsible for 
ensuring that the scoring procedures are con¬ 
sistent with the purpose(s) of the test and 
facilitate meaningful score interpretation. The 
nature of the intended score interpretations 


JA2282 


AERA_APA_NCME_0000049 


Case l:14-cv-00857-TSC Document 60-25 
USCA Case #17-7035 Document #1715850 


Filed 12/21/15 Page 51 of 100 
Filed: 01/31/2018 Page 524 of 573 


TEST DEVELOPMENT AND REVISION / PART I 


will determine the importance of psychometric 
characteristics of items in the test construction 
process. For example, indices of item difficulty 
and discrimination, and inter-item correlations, 
may be particularly important when relative 
score Interpretations are intended. In the case 
of relative score interpretations, good discrim¬ 
ination among test takers at all points along 
the construct continuum is desirable. It is 
important, however, that the test specifica¬ 
tions are not compromised when optimizing 
the distribution of these indices. In the case 
of absolute score interpretations, different cri¬ 
teria apply. In this case, the extent to which 
the relevant domain has been adequately rep¬ 
resented is important even if many of the 
items are relatively easy or nondiscriminating 
within a relevant population. It is important, 
however, to assure the quality of the content 
of relatively easy or nondiscriminating items. 
If cut scores are necessary for score interpreta¬ 
tion in criterion-referenced programs, the level 
of item discrimination constitutes critical 
information primarily in the vicinity of che 
cut scores. Because of these differences in test 
development procedures, tests designed to 
facilitate one type of interpretation function 
less effectively for other types of interpretation. 
Given appropriate test design and supporting 
evidence, however, scores arising from some 
norm-referenced programs may provide rea¬ 
sonable absolute score interpretations and 
scores arising from some criterion-refer¬ 
enced programs may provide reasonable rela¬ 
tive score interpretations. 

When evaluating the quality of the items 
in the item pool and the test itself, test devel¬ 
opers often conduct studies of differential 
item functioning (see chapter 7). Differencial 
item functioning is said to exist when test 
takers of approximately equal ability on the 
targeted construct or content domain differ 
in their responses to an item according to their 
group membership. In theory, the ultimate 
goal of such studies is to identify construct- 
irrelevant aspects of item content, item format, 


or scoring criteria that may differentially affect 
test scores of one or more groups of test tak¬ 
ers. When differential item functioning is 
detected, test developers try to identify plausi¬ 
ble explanations for the differences, and then 
they may replace or revise items that give rise 
to group differences if construct irrelevance is 
deemed likely. However, at this time, there has 
been little progress in discerning the cause or 
substantive themes that account for differen¬ 
tial item functioning on a group basis. Items 
for which the differential item functioning 
index is significant may constitute valid meas¬ 
ures of an element of the intended domain and 
differ in no way from other items that show 
nonsignificant indexes. When the differential 
item functioning index is significant, the test 
developer must take care that any replacement 
items or item revisions do not compromise 
the test specifications. 

When multiple forms of a test are pre¬ 
pared, the test specifications govern each of 
the forms. Also, when an item pool is devel¬ 
oped for a computerized adaptive test, the 
specifications refer both to the item pool and 
to the rules or procedures by which the indi¬ 
vidual item sets are created for each test taker. 
Some of the attractive features of computer¬ 
ized adaptive tests, such as tailoring the diffi¬ 
culty level of the items to the test takers 
ability, place additional constraints on che 
design of such tests. In general, a large num¬ 
ber of items is needed for a computerized 
adaptive test to ensure that each tailored item 
set meets the requirements of the test specifi¬ 
cations. Further, rests often are developed in 
the context of larger systems or programs. 
Multiple item sets, for example, may be creat¬ 
ed for use with different groups of test takers 
or on different testing dates. Last, when a 
short form of a test is prepared, the test speci¬ 
fications of che original test govern the short 
form. Differences in the test specifications 
and the psychometric properties of the short 
form and the original test will affect the inter¬ 
pretation of the scores derived from the short 
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form. In any of these cases, the same funda¬ 
mental methods and principles of test devel¬ 
opment apply. 

Special Considerations in Test 
Development 

This section elaborates on several topics dis¬ 
cussed above. First, considerations in delin¬ 
eating the framework for the test are discussed. 
Following this, considerations in the develop¬ 
ment of performance assessments and portfolios 
are addressed. 

Delineating the Framework for 
the Test 

The scenario presented above outlines what is 
often done to develop a test. However, the activ¬ 
ities do not always happen in a rigid sequence. 
There is often a subtle interplay between the 
process of conceptualizing a construct or con¬ 
tent domain and the development of a test of 
that construct or domain. The framework for 
the test provides a description of how the 
construct or domain will be represented. The 
procedures used to develop items and scoring 
rubrics and to examine item characteristics 
may often contribute to clarifying the frame¬ 
work. The extent to which the framework is 
defined a priori is dependent on the testing 
application. In many testing applications, a 
well-defined framework and detailed test speci¬ 
fications guide the development of items and 
their associated scoring rubrics and procedures. 
In some areas of psychological measurement, 
test development may be less dependent on 
an a priori defined framework and may rely 
more on a data-based approach that results in 
an empirically derived definition of the frame¬ 
work. In such instances, items are selected 
primarily on the basis of their empirical rela¬ 
tionship with an external criterion, their rela¬ 
tionships with one another, or their power to 
discriminate among groups of individuals. For 
example, construction of a selection test for 
sales personnel might be guided by the corre¬ 


lations of item scores with productivity meas¬ 
ures of current sales personnel or a measure of 
client satisfaction might be assembled from those 
items in an item pool that correlate most highly 
with customer loyalty. Similarly, an inventory 
to help identify different patterns of psychopa¬ 
thology might be developed using patients from 
different diagnostic subgroups. When test 
development relies on a data-based approach, 
it is likely that some items will be selected based 
on chance occurrences in the data. Cross-valida¬ 
tion studies are routinely conducted to deter¬ 
mine the tendency to select items by chance, 
which involves administering the test to a 
comparable sample. 

In many testing applications, the frame¬ 
work for the test is specified initially and this 
specification subsequently guides the develop¬ 
ment of items and scoring procedures. Empirical 
relationships may then be used to inform 
decisions about retaining, rejecting, or modi¬ 
fying items. Interpretations of scores from tests 
developed by this process have the advantage 
of a logical/theoretical and an empirical foun¬ 
dation for the underlying dimensions repre¬ 
sented by the test. 

Performance Assessments 

One distinction between performance 
assessments and other forms of tests has to do 
with the type of response that is required from 
the test takers. Performance assessments require 
the test takers to carry out a process such as 
playing a musical instrument or tuning a car’s 
engine or to produce a product such as a writ¬ 
ten essay. Performance assessments generally 
require the test takers to demonstrate their 
abilities or skills in settings that closely resem¬ 
ble real-life settings. For example, an assess¬ 
ment of a psychologist in training may require 
the test taker to interview a client, choose 
appropriate tests, and arrive at diagnosis and 
plan for therapy. Performance assessments are 
diverse in nature and can be product-based as 
well as behavior-based. Because performance 
assessments typically consist of a small num- 
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her of tasks, establishing the extent to which 
the results can be generalized to the broader 
domain is particularly important. The use of 
test specifications will contribute to tasks being 
cieveloped so as to systematically represent the 
critical dimensions to be assessed, leading to a 
more comprehensive coverage of the domain 
than what would occur if test specifications were 
not used. Further, both logical and empirical 
evidence are important to document the extent 
to which performance assessments—tasks as 
well as scoring criteria—reflect the processes 
or skills that are specified by the domain 
definition. When tasks are designed to elicit 
complex cognitive processes, logical analyses 
of the tasks and both logical and empirical 
analyses of the test takers’ performances on 
the tasks provide necessary validity evidence. 

Portfolios 

A unique type of performance assessment is an 
individual portfolio. Portfolios are systematic 
collections of work or educational products 
typically collected over time. Like other assess¬ 
ment procedures, the design of portfolios is 
dependent on the purpose. Typical purposes 
include judgment of the improvement in job 
or educational performance and evaluation of 
the eligibility for employment, promotion, or 
graduation. A well-designed portfolio specifies 
the nature of the work that is to be put into the 
portfolio. The portfolio may include entries such 
as representative products, the best work of the 
test taker, or indicators of progress. For example, 
in an employment setting involving promotion, 
employees may be instructed to include their 
best work or products. Alternatively, if the pur¬ 
pose is to judge a students educational growth, 
students may be asked to provide evidence of 
improvement with respect to particular com¬ 
petencies or skills. They may also be requested 
to provide justifications for the choices. Still other 
methods may include the use of videotapes, exhi¬ 
bitions, demonstrations, simulations, and so on. 

In employment settings, employees may be 
involved in the selection of their work and prod¬ 
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ucts that demonstrate their competencies for 
promotion purposes. Analogously, in educa¬ 
tional applications, students may participate in 
the selection of some of their work and the prod¬ 
ucts to be included in their portfolios as well as 
in the evaluation of the materials. The specifi¬ 
cations for the portfolio indicate who is respon¬ 
sible for selecting its contents. For example, the 
specifications may state that the test taker, the 
examiner, or both parties working together should 
be involved in the selection of the concerns of the 
portfolio. The particular responsibilities of each 
party are delineated in the specifications. The 
more standardized the contents and procedures 
of administration, the easier it is to establish 
comparability of portfolio-based scores. 
Regardless of the methods used, all performance 
assessments are evaluated by the same standards 
of technical quality as other forms of tests. 

Test Revisions 

Tests and their supporting documents (e.g., test 
manuals, technical manuals, users guides) are 
reviewed periodically to determine whether 
revisions are needed. Revisions or amendments 
are necessary when new research data, significant 
changes in the domain, or new conditions of 
cest use and interpretation would either improve 
the validity of interpretations of the test scores 
or suggest that the test is no longer fully appro¬ 
priate for its intended use. As an example, tests 
are revised if the test content or language has be¬ 
come outdated and, therefore, may subsequently 
affect the validity of the test score interpretations. 
Revisions to cest content are also made to ensure 
the confidentiality of the test. It should be noted, 
however, that outdated norms may not have the 
same implications for revisions as an outdated test. 
For example, it may be necessary co update the 
norms for an achievement test after a period of 
rising or falling achievement in the norming 
population, or when there are changes in the 
test-taking population, but the test content 
itself may continue to be as relevant as it was 
when the test was developed. 
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STANDARDS] 


Standard 3.1 

Tests and testing programs should be devel¬ 
oped on a sound scientific basis. Test devel¬ 
opers and publishers should compile and 
document adequate evidence bearing on 
test development. 

Standard 3.2 

The purpose(s) of the test, definition of the 
domain, and the test specifications should 
be stated clearly so that judgments can be 
made about the appropriateness of the 
defined domain for the stated purpose(s) 
of the test and about the relation of items 
to the dimensions of the domain they are 
intended to represent. 

Comment: The adequacy and usefulness of 
test interpretations depend on the rigor with 
which the purposes of the test and the domain 
represented by the test have been defined and 
explicated. The domain definition should be 
sufficiently detailed and delimited to show 
clearly what dimensions of knowledge, skill, 
processes, attitude, values, emotions, or 
behavior are included and what dimensions 
are excluded. A clear description will enhance 
accurate judgments by reviewers and others 
about the congruence of the defined domain 
and the test items. 

Standard 3.3 

The test specifications should be document¬ 
ed, along with their rationale and the 
process by which they were developed. The 
test specifications should define the content 
of the test, the proposed number of items, 
the item formats, the desired psychometric 
properties of the items, and the item and 
section arrangement. They should also speci¬ 
fy the amount of time for testing, directions 
to the test takers, procedures to be used for 
test administration and scoring, and other 
relevant information. 


Comment: Professional judgment plays a major 
role in developing the test specifications. The 
specific procedures used for developing the 
specifications depend on the purposes of the 
test. For example, in developing licensure and 
certification tests, practice analyses or job analy¬ 
ses usually provide the basis for defining the 
test specifications, and job analyses primarily 
serve this function for employment tests. For 
achievement tests to be given at the end of a 
course, the test specifications should be based 
on an outline of course content and goals. 
Whereas, for placement tests, it may be nec¬ 
essary to examine the required entry knowl¬ 
edge and skills for several courses. 

Standard 3.4 

The procedures used to interpret test scores, 
and, when appropriate, the normative or 
standardization samples or the criterion used 
should be documented. 

Comment: Test specifications may indicate that 
the intended score interpretations are for absolute 
or relative score interpretations, or both. In rel¬ 
ative score interpretations the status of an indi¬ 
vidual (or group) is determined by comparing 
the score (or mean score) to the performance of 
others in one or more defined populations. In 
absolute score interpretations, the score or aver¬ 
age is assumed to reflect directly a level of com¬ 
petence or mastery in some defined criterion 
domain. Tests designed to facilitate one type of 
interpretation function less effectively for other 
types of interpretations. Given appropriate test 
design and adequate supporting daca, however, 
scores arising from norm-referenced testing pro¬ 
grams may provide reasonable absolute score 
interpretations and scores arising from criterion- 
referenced programs may provide reasonable 
relative score interpretations. 

Standard 3.5 

When appropriate, relevant experts external 
to the testing program should review the test 
specifications. The purpose of the review, the 
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process by which the review is conducted, 
and the results of the review should be docu¬ 
mented. The qualifications, relevant experi¬ 
ences, and demographic characteristics of 
expert judges should also be documented. 

Comment: Expert review of the test specifica¬ 
tions may serve many useful purposes such as 
helping to assure content quality and repre¬ 
sentativeness. The expert judges may include 
individuals representing defined populations 
of concern to the test specifications. For exam¬ 
ple, if the test is related to ethnic minority 
concerns, the expert review typically includes 
members of appropriate ethnic minority 
groups or experts on minority group issues. 

Standard 3.6 

The type of items, the response formats, scor¬ 
ing procedures, and test administration proce¬ 
dures should be selected based on the purposes 
of the test, the domain to be measured, and 
the intended test takers. To the extent possible, 
test content should be chosen to ensure that 
intended inferences from test scores are equally 
valid for members of different groups of test 
takers. The test review process should include 
empirical analyses and, when appropriate, the 
use of expert judges to review items and 
response formats. The qualifications, relevant 
experiences, and demographic characteristics 
of expert judges should also be documented. 

Comment: Expert judges may be asked to iden¬ 
tify material likely to be inappropriate, confus¬ 
ing, or offensive for groups in the test-taking 
population. For example, judges may be asked 
to identify whether lack of exposure to problem 
contexts in mathematics word problems may 
be of concern for some groups of students. 
Various groups of test takers can be defined by 
characteristics such as age, ethnicity, culture, 
gender, disability, or demographic region. 
There is limited evidence, however, that expert 
reviews alleviate problems with bias in testing 
(see chapter 7). 
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Standard 3.7 

The procedures used to develop, review, and 
try out items, and to select items from the 
item pool should be documented. If the 
items were classified into different categories 
or subtests according to the test specifica¬ 
tions, the procedures used for the classifica¬ 
tion and the appropriateness and accuracy 
of the classification should be documented. 

Comment: Empirical evidence and/or expert 
judgment are used to classify items according 
to categories of the test specifications. For 
example, professional panels may be used for 
classifying the items or for determining the 
appropriateness of the developer’s classifica¬ 
tion scheme. The panel and procedures used 
should be chosen with care as they will affect 
the accuracy of the classification. 

Standard 3.8 

When item tryouts or field tests are con¬ 
ducted, the procedures used to select the 
sample(s) of test takers for item tryouts and 
the resulting characteristics of the sample(s) 
should be documented. When appropriate, 
the sample(s) should be as representative as 
possible of the population(s) for which the 
test is intended. 

Comment: Conditions which may differential¬ 
ly affect performance on the test items by the 
sample(s) as compared to the intended popu- 
lation(s) should be documented when appro¬ 
priate. As an example, test takers may be less 
motivated when they know their scores will 
not have an impact on them. 

Standard 3.9 

When a test developer evaluates the psycho¬ 
metric properties of items, the classical or 
item response theory (IRT) model used for 
evaluating the psychometric properties of 
items should be documented. The sample used 
for estimating item properties should be de- 
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scribed and should be of adequate size and diver¬ 
sity for the procedure. The process by which 
items are selected and the data used for item 
selection, such as item difficulty, item discrimi¬ 
nation, and/or item information, should also 
be documented. When IRT is used to estimate 
item parameters in test development, the item 
response model, estimation procedures, and 
evidence of model fit should be documented. 

Comment: Although overall sample size is 
important, it is important also that there be an 
adequate number of cases in regions critical to 
the determination of the psychometric proper¬ 
ties of items. If the test is to achieve greatest 
precision in a particular part of the score scale 
and this consideration affects item selection, 
the manner in which item statistics are used 
needs to be carefully described. When IRT is 
used as the basis of test development, it is 
important to document the adequacy of fit of 
the model to the data. This is accomplished by 
providing information about the extent to 
which IRT assumptions (e.g., unidimensionali¬ 
ty, local item independence, or equality of slope 
parameters) are satisfied. 

Test developers should show that any dif¬ 
ferences between the administration conditions 
of the field test and the final form do not affect 
item performance. Conditions that can affect 
item statistics include item position, time 
limits, length of test, mode of testing (e.g., 
paper-and-pencil versus computer-administered), 
and use of calculators or other cools. For exam¬ 
ple, in field testing items, those placed at the 
end of a test might obtain poorer item statis¬ 
tics than chose inserted within the test. 

Standard 3.10 

Test developers should conduct cross-valida¬ 
tion studies when items are selected primari¬ 
ly on the basis of empirical relationships 
rather than on the basis of content or theoreti¬ 
cal considerations. The extent to which the dif¬ 
ferent studies identify the same item set should 
be documented. 


Comment: When dara-based approaches to test 
development are used, items are selected prima¬ 
rily on the basis of their empirical relationships 
with an external criterion, their relationships 
with one another, or their power to discrimi¬ 
nate among groups of individuals. Under these 
circumstances, it is likely that some items will 
be selected based on chance occurrences in the 
data used. Administering the test to a compara¬ 
ble sample of test takers or a hold-out sample 
provides a means by which the tendency to 
select items by chance can be determined. 

Standard 3.11 

Test developers should document the extent to 
which the content domain of a test represents 
the defined domain and test specifications. 

Comment: Test developers should provide evi¬ 
dence of the extent to which the test items and 
scoring criteria represent the defined domain. This 
affords a basis to help determine whether per¬ 
formance on the test can be generalized to the 
domain that is being assessed. This is especially 
important for tests that contain a small number 
of items such as performance assessments. Such 
evidence may be provided by expert judges. 

Standard 3.12 

The rationale and supporting evidence for 
computerized adaptive tests should be docu¬ 
mented. This documentation should include 
procedures used in selecting subsets of items 
for administration, in determining the start¬ 
ing point and termination conditions for the 
test, in scoring the test, and for controlling 
item exposure. 

Comment: It is important to assure that docu¬ 
mentation of the procedures does not com¬ 
promise the security of the test items. 

If a computerized adaptive test is intended 
to measure a number of different content sub¬ 
categories, item selection procedures are to assure 
that the subcategories are adequately represented 
by the items presented to the test taker. 
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Standard 3.13 

When a test score is derived from the differen¬ 
tial weighting of items, the test developer 
should document the rationale and process used 
to develop, review, and assign item weights. 
When the item weights are obtained based on 
empirical data, the sample used for obtaining 
item weights should be sufficiently large and 
representative of the population for which the 
test is intended. When the item weights are 
obtained based on expert judgment, the quali¬ 
fications of the judges should be documented. 

Comment: Changes in che population of test 
takers, along with other changes such as changes 
in instructions, training, or job requirements, 
may impact the original derived item weights, 
necessitating subsequent srudies after an 
appropriate period of time. 

Standard 3.14 

The criteria used for scoring test takers’ per¬ 
formance on extended-response items should be 
documented. This documentation is especially 
important for performance assessments, such as 
scorable portfolios and essays, where the criteria 
for scoring may not be obvious to the user. 

Comment: The completeness and clarity of the 
test specifications, including the definition of the 
domain, are essential in developing the scoring 
criteria. The test developer needs to provide a 
clear description of how the test scores are 
intended to be interpreted to help ensure the 
appropriateness of the scoring procedures. 

Standard 3.15 

When using a standardized testing format to 
collect structured behavior samples, the domain, 
test design, test specifications, and materials 
should be documented as for any other test. 
Such documentation should include a clear 
definition of the behavior expected of the test 
takers, the nature of the expected responses, and 
any materials or directions that are necessary 
to carry out the testing. 


Comment: In dr/eloping a prompt, the age, lan¬ 
guage, experience, and ability level of test takers 
should be considered, as should other possible 
unique sources of difficulty for groups in the po¬ 
pulation to be tested. Test directions that specify 
time allowances, nature of the responses expect¬ 
ed, and rules regarding use of supplementary 
materials, such as notes, references, dictionaries, 
calculators, or manipuiatives such as lab equip¬ 
ment, may be established via field testing. 

Standard 3.16 

If a short form of a test is prepared, for exam¬ 
ple, by reducing the number of items on the 
original test or organizing portions of a test into 
a separate form, the specifications of the short 
form should be as similar as possible to those 
of the original test. The procedures used for 
the reduction of items should be documented. 

Comment: The extent to which the specifica¬ 
tions of the short form differ from those of 
the original test, and the implications of such 
differences for interpreting the scores derived 
from"the short form, should be documented. 

Standard 3.17 

When previous research indicates that irrele¬ 
vant variance could confound the domain def¬ 
inition underlying the test, then to the extent 
feasible, the test developer should investigate 
sources of irrelevant variance. Where possible, 
such sources of irrelevant variance should be 
removed or reduced by the test developer. 

Standard 3.18 

For tests that have time limits, test development 
research should examine the degree to which 
scores include a speed component and evaluate 
the appropriateness of that component, given 
the domain the test is designed to measure. 

Standard 3.19 

The directions for test administration should 
be presented with sufficient clarity and empha- 
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sis so that it is possible for others to replicate 
adequately the administration conditions under 
which the data on reliability and validity, and, 
where appropriate, norms were obtained. 

Comment: Because all people administering 
tests, including those in schools, industry, and 
clinics, need to follow test administration con¬ 
ditions carefully, it is essential that test admin¬ 
istrators receive detailed instructions on rest 
administration guidelines and procedures. 

Standard 3.20 

The instructions presented to test takers should 
contain sufficient detail so that test takers can 
respond to a task in the manner that the test 
developer intended. When appropriate, sample 
material, practice or sample questions, criteria 
for scoring, and a representative item identi¬ 
fied with each major area in the test’s classifi¬ 
cation or domain should be provided to the 
test takers prior to the administration of the 
test or included in the testing material as pan 
of the standard administration instructions. 

Comment: For example, in a personality 
inventory it may be intended chat test takers 
give the first response that occurs to them. 
Such an expectation should be made clear in 
the inventory directions. As another example, 
in directions for interest or occupational 
inventories, it may be important to specify 
whether test takers are to mark the activities 
they would like ideally or whether they are 
to consider both their opportunity and their 
ability realistically. 

The extent and nature of practice materi¬ 
als and directions depend on expected levels 
of knowledge among test takers. For example, 
in using a novel test format, it may be very 
important to provide the test taker a practice 
opportunity as part of the test administration. 
In some testing situations, it may be important 
for the instructions to address such matters as 
the effects that guessing and time limits have 
on test scores. If expansion or elaboration of 
the test instructions is permitted, the condi¬ 


tions under which this may be done should be 
stated clearly in the form of general rules and 
by giving representative examples. If no expan¬ 
sion or elaboration is to be permitted, this 
should be stated explicidy. Publishers should 
include guidance for dealing with typical 
questions from test takers. Users should be 
instructed how to deal with questions that 
may arise during the testing period. 

Standard 3.21 

If the test developer indicates that the condi¬ 
tions of administration are permitted to vary 
from one test taker or group to another, per¬ 
missible variation in conditions for adminis¬ 
tration should be identified, and a rationale 
for permitting the different conditions should 
be documented. 

Comment: In deciding whether the conditions 
of administration can vary, the test developer 
needs to consider and study the potential 
effects of varying conditions of administra¬ 
tion. If conditions of administration vary 
from the conditions studied by the test devel¬ 
oper or from those used in the development 
of norms, the comparability of the test scores 
may be weakened and the applicability of the 
norms can be questioned. 

Standard 3.22 

Procedures for scoring and, if relevant, 
scoring criteria should be presented by 
the test developer in sufficient detail and 
clarity to maximize the accuracy of scoring. 
Instructions for using rating scales or for 
deriving scores obtained by coding, scaling, 
or classifying constructed responses should 
be clear. This is especially critical if tests 
can be scored locally. 

Standard 3.23 

The process for selecting, training, and qualify¬ 
ing scorers should be documented by the test 
developer. The training materials, such as the 
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scoring rubrics and examples of test takers’ 
responses that illustrate the levels on the score 
scale, and the procedures for training scorers 
should result in a degree of agreement among 
scorers that allows for the scores to be interpret¬ 
ed as originally intended by the test developer. 
Scorer reliability and potential drift over time 
in raters’ scoring standards should be evaluat¬ 
ed and reported by the person(s) responsible 
for conducting the training session. 

Standard 3.24 

When scoring is done locally and requires 
scorer judgment, the test user is responsible 
for providing adequate training and instruc¬ 
tion to the scorers and for examining scoter 
agreement and accuracy. The test developer 
should document the expected level of scorer 
agreement and accuracy. 

Comment: A common practice of test devel¬ 
opers is to provide examples of training mate¬ 
rials (e.g., scoring rubrics, rest takers’ responses 
at each score level) and procedures when scoring 
is done locally and requires scorer judgment. 

Standard 3.25 

A test should be amended or revised when 
new research data, significant changes in the 
domain represented, or newly recommended 
conditions of test use may lower the validity 
of test score interpretations. Although a test 
that remains useful need not be withdrawn 
or revised simply because of the passage of 
time, test developers and test publishers are 
responsible for monitoring changing condi¬ 
tions and for amending, revising, or with¬ 
drawing the test as indicated. 

Comment: Test developers need to consider a 
number of factors that may warrant the revi¬ 
sion of a test, including outdated tesc content 
and language. If an older version of a test is 
used when a newer version has been published 
or made available, test users are responsible for 


providing evidence chat the older version is 
as appropriate as the new version for that 
particular test use. 

Standard 3.26 

Tests should be labeled or advertised as 
“revised” only when they have been revised 
in significant ways. A phrase such as "with 
minor modification” should be used when 
the test has been modified in minor ways. 
The score scale should be adjusted to account 
for these modifications, and users should be 
informed of the adjustments made to the 

Comment: It is the test developers responsi¬ 
bility to determine whether revisions to a tesc 
would influence test score interpretations. If 
test score interpretations would be affected 
by the revisions, it would then be appropriate 
to label the test “revised.” When tests are 
revised, the nature of the revisions and their 
implications on test score interpretations 
should be documented. 

Standard 3.27 

If a test or part of a test is intended for 
research use only and is not distributed for 
operational use, statements to this effect 
should be displayed prominently on all rele¬ 
vant test administration and interpretation 
materials that are provided to the test user. 

Comment: This standard refers to tests that 
are intended for research use only and does 
not refer to standard test development func¬ 
tions chat occur prior to the operational use 
of a test (e.g., field testing). 
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Background 

Test scores are reported on scales designed to 
assist score interpretation. Typically, scoring 
begins with responses to separate test items, 
which are often coded using 0 or 1 to represent 
wrong/right or negative/positive, but sometimes 
using numerical values to indicate finer response 
gradations. Then the item scores are combined, 
often by addition but sometimes by a more 
elaborate procedure, to obtain a raw score. Raw 
scores are determined, in part, by features of a 
test such as test length, choice of time limit, 
item difficulties, and the circumstances under 
which the test is administered. This makes raw 
scores difficult to interpret in the absence of 
further information. Interpretation and statisti¬ 
cal analyses may be facilitated by converting 
raw scores into an entirely different set of val¬ 
ues called derived, scores or scale scores. The vari¬ 
ous scales used for reporting scores on college 
admissions tests, the standard scores often 
used to report results for intelligence scales or 
vocational interest and personality inventories, 
and the grade equivalents reported for achieve¬ 
ment tests in the elementary grades are exam¬ 
ples of scale scores. The process of developing 
such a score scale is called scaling a test. Scale 
scores may aid interpretation by indicating 
how a given score compares to those of other 
test takers, by enhancing the comparability of 
scores obtained using different forms of a test, 
or in other ways. 

Another way of assisting score interpreta¬ 
tion is to establish standards or cut scores that 
distinguish different score ranges. In some 
cases, a single cut score may define the bound¬ 
ary between passing and failing. In other cases, 
a series of cut scores may define distinct pro¬ 
ficiency levels. Cut scores may be established 
for either raw or scale scores. Both scale scores 
and standards or cut scores can be central to 
the use and interpretation of test scores. For 


that reason, their defensibility is an important 
consideration in test validation. There is a close 
connection between standards or cut scores 
and certain scale scores. If the successive score 
ranges defined by a series of cut scores are 
relabeled, say 0, 1,2, and so on, then a scale 
score has been created. 

In addition to facilitating interpretations 
of a single test form considered in isolation, 
scale scores are often created to enhance com¬ 
parability across different forms of the same 
test, across test formats or administration 
conditions, or even across tests designed to 
measure different constructs (e.g., related sub¬ 
tests in a battery). Equated scores from alter¬ 
nate forms of a test can often be interpreted 
mote easily when expressed in scale score units 
rather than raw score units. Scaling may be 
used to place scores from different levels of an 
achievement test on a continuous scale and 
thereby facilitate inferences about growth or 
development. Scaling can also enhance the 
comparability of scores derived from tests in 
different areas, as in subtests within an apti¬ 
tude, interest, or achievement battery. 

Norm-Referenced and Criterion- 
Referenced Score Interpretations 

Individual raw scores or scale scores are often 
referred to the distribution of scores for one 
or more comparison groups to draw useful 
inferences about an individuals performance. 
Test score interpretations based on such compar¬ 
isons are said to be norm-referenced Percentile 
rank norms, for example, indicate the stand¬ 
ing of an individual or group within a defined 
population of individuals or groups. An example 
of such a comparison group might be fourth- 
grade students in the United States, tested in 
the last 2 months of a recent school year. 
Percentiles, averages, or other statistics for such 
reference groups are called norms. By showing 
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how the test score of a given examinee com¬ 
pares to those of others, norms assist in the 
classification or description of examinees. 

Other test score interpretations make no 
direct reference to the performance of other 
examinees. These interpretations may take a 
variety of forms; most are collectively referred 
to as criterion-referenced interpretations. Derived 
scores supporting such interpretations may 
indicate the likely proportion of correct 
responses on some larger domain of items, or 
the probability of an examinee’s answering 
particular sorts of items correctly. Other crite¬ 
rion-referenced interpretations may indicate 
the likelihood that some psychopathology is 
present. Still other criterion-referenced inter¬ 
pretations indicate the probability that an 
examinees level of tested knowledge or skill 
is adequate to perform successfully in some 
other setting; such probabilities may be sum¬ 
marized in an expectancy table. Scale scores 
to support such criterion-referenced score 
interpretations are often developed on the 
basis of statistical analyses of the relationships 
of test scores to other variables. 

Some scale scores are developed primarily 
to support norm-referenced interpretations 
and others, criterion-referenced interpretations. 
In practice, however, there is not always a sharp 
distinction. Both criterion-referenced and 
norm-referenced scales may be developed and 
used for the same cest scores. Moreover, a 
norm-referenced score scale originally devel¬ 
oped, for example, to indicate performance 
relative to some specific reference population 
might, over time, also come to support crite¬ 
rion-referenced interpretations. This could 
happen as research and experience brought 
increased understanding of the capabilities 
implied by different scale score levels. 
Conversely, results of an educational assess¬ 
ment might be reported on a scale consisting 
of several ordered proficiency levels, defined 
by descriptions of the kinds of tasks students 
at each level were able to perform. That would 
be a criterion-referenced scale, but once the 
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distribution of scores over levels was reported, 
say, for all eighth-grade students in a given 
state, individual students’ scores would also 
convey information about their standing rela- 
. tive to that tested population. 

Interpretations based on cut scores may 
likewise be either criterion-referenced or 
norm-referenced. If qualitatively different 
descriptions are attached to successive score 
ranges, a criterion-referenced interpretation is 
supported. For example, the descriptions of 
performance levels in some assessment task 
scoring rubrics can enhance score interpreta¬ 
tion by summarizing the capabilities that must 
be demonstrated to merit a given score. In 
other cases, criterion-referenced interpretations 
may be based on empirically determined rela¬ 
tionships between test scores and other vari¬ 
ables. But when tests are used for selection, it 
may be appropriate to rank-order examinees 
according to their test performance and estab¬ 
lish a cut score so as to select a prespecified 
number or proportion of examinees from one 
end of the distribution, if the selection use is 
otherwise supported by relevant reliability 
and validity evidence. In such cases, the cut 
score interpretation is norm-referenced; the 
labels reject or fail versus accept or pass are 
determined solely by an examinee’s standing 
relative to others tested. 

Criterion-referenced interpretations based 
on cut scores are sometimes criticized on the 
grounds that there is very rarely a sharp dis¬ 
tinction of any kind between those just below 
versus just above a cut score. A neuropsy¬ 
chological rest may be helpful in diagnosing 
some particular impairment, for example, but 
the probability that the impairment is pres¬ 
ent is likely to increase continuously as a 
function of the test score. Cut scores may 
nonetheless aid in formulating rules for 
reaching decisions on the basis of test per¬ 
formance. It should be recognized, however, 
that the probability of misclassification will 
generally be relatively high for persons with 
scores close to the cut points. 
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Norms 

The validity of norm-referenced interpretations 
depends in part on the appropriateness of the 
reference group co which test scores are com¬ 
pared. Norms based on hospitalized patients, 
for example, might be inappropriate for some 
interpretations of nonhospitalized patients’ 
scores. Thus, it is important that reference 
populations be carefully„defined and clearly 
described. Validity of such interpretations also 
depends on the accuracy with which norms 
summarize the performance of the reference 
population. That population may be small 
enough that essentially the entire population 
can be tested (e.g., all pupils at a given grade 
level in a given district tested on the same 
occasion). Often, however, only a sample of 
examinees from the reference population is 
tested. It is then important that the norms be 
based on a technically sound, representative, 
scientific sample of sufficient size. Patients in 
a few hospitals in a small geographic region 
are unlikely to be representative of all patients 
in the United States, for example. Moreover, 
the appropriateness of norms based on a given 
sample may diminish over time. Thus, for tests 
that have been in use for a number of years, 
periodic review is generally required to assure 
the continued utility of norms. Renorming may 
be required to maintain the validity of norm- 
referenced test score interpretations. 

More than one reference population may 
be appropriate for the same test. For example, 
achievement test performance might be inter¬ 
preted by reference to local norms based on 
sampling from a particular school district, 
norms for a state or type of community, or 
national norms. For other tests, norms might 
be based on occupational or educational clas¬ 
sifications. Descriptive statistics for all exam¬ 
inees who happen to be tested during a given 
period of time (sometimes called user norms 
or program norms) may be useful for some 
purposes, such as describing trends over time. 
But there must be sound reason to regard that 


group of test takers as an appropriate basis for 
such inferences. When there is a suitable ration¬ 
ale for using such a group, the descriptive sta¬ 
tistics should be clearly characterized as being 
based on a sample of persons routinely tested 
as part of an ongoing program. 

Comparability and Equating 

Many test uses involve different versions of 
the same test, which yield scores that can be 
used interchangeably even though they are 
based on different sets of items. In testing 
programs that offer a choice of examination 
dates, for example, test security may be com¬ 
promised if the same form is used repeatedly. 
Other testing applications may entail repeated 
measurements of the same individuals, perhaps 
to measure change in levels of psychological 
dysfunction, change in attitudes, or educa¬ 
tional progress. In such contexts, reuse of the 
same set of test items may result in correlated 
errors of measurement and biased estimates 
of change. When distinct forms of a test ate 
constructed to the same explicit content and 
statistical specifications and administered 
under identical conditions, they are referred 
to as alternate forms or sometimes parallel or 
equivalent forms. The process of placing scores 
from such alternate forms on a common scale 
is called equating. Equating is analogous to 
the calibration of different balances so that 
they all indicate the same weight for any given 
object. However, the equating process for test 
scores is more complex. It involves small statis¬ 
tical adjustments to account for minor differ¬ 
ences in the difficulty and statistical properties 
of the alternate forms. 

In theory, equating should provide accu¬ 
rate score conversions for any set of persons 
drawn from the examinee population for which 
the test is designed. Furthermore, the same 
score conversion should be appropriate regard¬ 
less of the score interpretation or use intend¬ 
ed. It is not possible to construct conversions 
with these ideal properties between scores on 
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tests that measure different constructs; that 
differ materially in difficulty, reliability, time 
limits, or other conditions of administration; 
or that are designed to different specifications. 

There is another assessment approach 
that may provide interchangeable scores based 
on responses to different items using different 
methods, not referred to as equating. This is 
the use of adaptive tests. It has long been rec¬ 
ognized that little is learned from examinees' 
responses to items that are much too easy or 
much too difficult for them. Consequently, 
some testing procedures use only a subset of 
the available items with each examinee in 
order to avoid boredom or frustration, or to 
shorten testing time. An adaptive test con- 
sisrs of a pool of items together with rules 
for selecting a subset of those items to be 
administered to an individual examinee, and 
a procedure for placing different examinees’ 
scores on a common scale. The selection 
of successive items is based in part on the 
examinees responses to previous items. The 
item pool and item selection rules may be 
designed so that each examinee receives a 
representative set of items, of appropriate 
difficulty. The selection rules generally 
assure that an acceptable degree of precision 
is attained before testing is terminated. At 
one time, such tailored testing was limited 
to certain individually administered psy¬ 
chological tests. With advances in item 
response theory (IRT) and in computer 
technology, however, adaptive testing is 
becoming more sophisticated. With some 
adaptive tests, it may happen that two 
examinees rarely if ever respond to precisely 
the same set of items. Moreover, two exam¬ 
inees taking the same adaptive test may be 
given sets of items that differ markedly in 
difficulty. Nevertheless, when certain statis¬ 
tical and content conditions are met, test 
scores produced by an adaptive testing sys¬ 
tem can function like scores from equated 
alternate forms. 


Scaling to Achieve Comparability 

The term equating is properly reserved only 
for score conversions derived for alternate forms 
of the same test. It is often useful, however, to 
compare scores from tests that cannot, in the¬ 
ory, be equated. For example, it may be desir¬ 
able to interpret scores from a shortened (and 
hence less reliable) form of a test by first con¬ 
verting them to corresponding scores on the 
full-length version. For the evaluation of exam¬ 
inee growth over time, it may be desirable to 
develop scales that span a broad range of devel¬ 
opmental or educational levels. Test revision 
often brings a need for some linkage between 
scores obtained using newer and older editions. 
International comparative studies or use with 
hearing-impaired examinees may require test 
forms in different languages. In still other 
cases, linkages or alignments may be created 
between tests measuring different constructs, 
perhaps comparing an aptitude with a form 
of behavior, or linking measures of achieve¬ 
ment in several content areas. Scores from 
such tests may sometimes be aligned or pre¬ 
sented in a concordance table to aid users in 
estimating relative performance on one test 
from performance on another. 

Score conversions to facilitate such com¬ 
parisons may be described using terms like 
linkage, calibration, concordance, projection, 
moderation, or anchoring. These weaker score 
linkages may be technically sound and may 
fully satisfy desired goals of comparability for 
one purpose or for one subgroup of examinees, 
but they cannot be assumed to be stable over 
time or invariant across multiple subgroups of 
the examinee population nor is there any assur¬ 
ance that scores obtained using different tests 
will be equally accurate. Thus, their use for other 
purposes or with other populations than origi¬ 
nally intended may require additional research. 
For example, a score conversion that was accu¬ 
rate for a group of native speakers might sys¬ 
tematically overpredict or underpredict the 
scores of a group of nonnative speakers. 
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Cut Scores 

A critical step in the development and use of 
some tests is to establish one or more cut points 
dividing the score range to partition the dis¬ 
tribution of scores into categories. These cate¬ 
gories may be used just for descriptive purposes 
or may be used to distinguish among exam¬ 
inees for whom different programs are deemed 
desirable or different predictions are warrant¬ 
ed. An employer may determine a cut score 
to screen potential employees or promote cur¬ 
rent employees; a school may use test scores 
to decide which of several alternative instruc¬ 
tional programs would be most beneficial for 
a student; in granting a professional license, a 
state may specify a minimum passing score 
on a licensure test. 

These examples differ in important 
respects, but all involve delineating categories 
of examinees on the basis of test scores. Such 
cut scores embody the rules according to which 
tests are used or interpreted. Thus, in some 
situations the validity of test interpretations 
may hinge on the cut scores. There can be no 
single method for determining cut scores for 
all tests or for all purposes, nor can there be 
any single set of procedures for establishing 
their defensibility. These examples serve only 
as illustrations. 

The first example, that of an employer 
hiring all those who earn scores above a given 
level on an employment test, is most straight¬ 
forward. Assuming that the employment test 
is valid for its intended use, average job per¬ 
formance would typically be expected to rise 
steadily, albeit slowly, with each increment in 
test score, at least for some range of scores 
surrounding the cut point. In such a case the 
designation of the particular value for the cut 
point may be largely determined by the num¬ 
ber of persons to be hired or promoted. There 
is no sharp difference between those just below 
the cut point and those just above it, and the 
use of the cue score does not entail any crite¬ 
rion-referenced interpretation. This method 


of establishing a cut score may be subject to 
legal requirements with respect to the nature 
of the validity and reliability evidence needed 
to support the use of rank-order selections 
and the unavailability of effective alternative 
selection methods, if it has a disproportionate 
effect on one or more subgroups of employees 
or prospective employees. 

In the second example, a school district 
might structure its courses in writing around 
three categories of needs. For children whose 
proficiency is least developed, instruction 
might be provided in small groups, with con¬ 
siderable individual attention to assist them 
in creating meaningful written stories grounded 
in their own experience. For children whose 
proficiency was further developed, more empha¬ 
sis might be placed on systematic exploration 
of the stages of the writing process. Instruction 
for children at the highest proficiency level might 
emphasize mastery of specific writing genres 
or prose structures used in mote formal writ¬ 
ing. In an appropriate implementation of such 
a program, children could easily be transferred 
from one level to another if their original 
placement was in error or as their proficiency 
increased. Ideally, cut scores delineating cate¬ 
gories in this application would be based on 
research demonstrating empirically that pupils 
in successive score ranges did most often ben¬ 
efit more from the respective treatments to 
which they were assigned chan from the alter¬ 
natives available. It would typically be found 
that between those score ranges in which one 
or another instructional treatment was clearly 
superior, there was an intermediate region in 
which neither treatment was clearly preferred. 
The cut score might be located somewhere in 
that intermediate region. 

In the final example, that of a professional 
licensure examination, the cut score represents 
an informed judgment that those scoring below 
it are likely to make serious errors for want of 
the knowledge or skills tested. Litde evidence 
apart from errors made on the test itself may 
document the need to deny the right to prac- 
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tice the profession. No test is perfect, of 
course, and regardless of the cut score chosen, 
some examinees with inadequate skills are 
likely to pass and some with adequate skills 
are likely to fail. The relative probabilities of 
such false positive and false negative errors 
will vary depending on the cut score thosen. 
A given probability of exposing rhe public 
to potential harm by issuing a license to an 
incompetent individual (false positive) must 
be weighed against some corresponding 
probability of denying a license to, and there¬ 
by disenfranchising, a qualified examinee 
(false negative). Changing the cut score to 
reduce either probability will increase the 
other, although boch kinds of errors can be 
minimized through sound test design that 
anticipates the role of the cut score in test use 
and interpretation. Determining cut scores 
in such situations cannot be a purely tech¬ 
nical matter, although empirical studies 
and statistical models can be of great value 
in informing the process. 

Cut scores embody value judgments as 
well as technical and empirical considerations. 
Where the results of the standard-setting process 
have highly significant consequences, and 
especially where large numbers of examinees 
are involved, those responsible for establish¬ 
ing cut scores should be concerned that the 
process by which cut scores are determined be 
clearly documented and defensible. The qual¬ 
ifications of any judges involved in standard 
setting and the process by which chey are 
selected are part of that documentation. Care 
must be taken co assure chat judges under¬ 
stand what they are to do. The process musr 
be such that well-qualified judges can apply 
their knowledge and experience to reach 
meaningful and relevant judgments chat accu¬ 
rately reflect their understandings and inten¬ 
tions. A sufficiently large and representative 
group of judges should be involved to provide 
reasonable assurance that results would not 
vary greatly if the process were replicated. 


Standard 4.1 

Test documents should provide test users 
with clear explanations of the meaning and 
intended interpretation of derived score scales, 
as well as their limitations. 

Comment All scales (raw score or derived) may 
be subject to misinterpretation. Sometimes 
scales are extrapolated beyond the range of 
available data or ate interpolated without suffi¬ 
cient data points. Grade- and age-equivalent 
scores have been criticized in this regard, but 
percentile ranks and standard score scales are 
also subject to misinterpretation. If the nature 
or incended uses of a scale are novel, ic is espe¬ 
cially important that its uses, interpretations, 
and limitations be clearly described. Illustrations 
of appropriate versus inappropriate interpreta¬ 
tions may be helpful, especially for types of 
scales or interpretations that may be unfamiliar 
to most users. This standard pertains to score 
scales intended for criterion-referenced as well 
as for norm-referenced interpretation. 

Standard 4.2 

The construction of scales used for report¬ 
ing scores should be described clearly in 
test documents. 

Comment: When scales, norms, or other 
interpretive systems are provided by the test 
developer, technical documentation should 
enable users to judge the quality and preci¬ 
sion of the resulting derived scores. This 
standard pertains to score scales intended for 
criterion-referenced as well as for norm-refer¬ 
enced interpretation. 

Standard 4.3 

If there is sound reason to believe that spe¬ 
cific misinterpretations of a score scale are 
likely, test users should be explicitly fore¬ 
warned. 
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Comment: Test publishers and users can reduce 
misinterpretations of grade-equivalent scores, 
for example, by ensuring that such scores are 
accompanied by instructions that make clear 
that grade-equivalent scores do not represent a 
standard of growth per year or grade and that 
roughly 50% of the students tested in the stan¬ 
dardization sample should by definition fall 
below grade level. As another example, a score 
scale point originally defined as the mean of 
some reference population should no longer be 
interpreted as representing average perform¬ 
ance if the scale is held constant over time and 
the examinee population changes. 

Standard 4.4 

When raw scores are intended to be directly 
interpretable, their meanings, intended 
interpretations, and limitations should be 
described and justified in the same manner 
as is done for derived score scales. 

Comment: In some cases the items in a test 
are a representative sample of a well-defined 
domain of items. The proportion correct on 
the test may then be interpreted as an estimate 
of the proportion of items in the domain that 
could be answered correctly. In other cases, 
different interpretations may be attached to 
scores above or below one or another cut score. 
Support should be offered for any such inter¬ 
pretations recommended by the test developer. 

Standard 4.5 

Norms, if used, should refer to clearly 
described populations. These populations 
should include individuals or groups to 
whom test users will ordinarily wish to 
compare their own examinees. 

Comment: It is the responsibility of test develop¬ 
ers to describe norms clearly and the responsibil¬ 
ity of test users to employ norms appropriately. 
Users need to know the applicability of a test to 
different groups. Differentiated norms or sum¬ 


mary information about differences between 
gender, ethnic, language, disability, grade, or 
age groups, for example, may be useful in some 
cases. The permissible uses of such differenti¬ 
ated norms and related information may be 
limited by law. Users also need to be made alert 
to situations in which norms are less appropri¬ 
ate for some groups or individuals than others. 
On an occupational interest inventory, for 
example, norms for persons actually engaged 
in an occupation may be inappropriate for 
interpreting the scores of persons not so 
engaged. As another example, the appropri¬ 
ateness of norms for personality inventories 
or relationship scales may differ depending 
upon an examinees sexual orientation. 

Standard 4.6 

Reports of norming studies should include 
precise specification of the population that 
was sampled, sampling procedures and par¬ 
ticipation rates, any weighting of the sample, 
the dates of testing, and descriptive statistics. 
The information provided should be sufficient 
to enable users to judge the appropriateness of 
the norms for interpreting the scores of local 
examinees. Technical documentation should 
indicate the precision of the norms themselves. 

Comment: Scientific sampling is important if 
norms are to be representative of intended 
populations. For example, schools already 
using a given published test and volunteering 
to participate in a norming study for that test 
should not be assumed to be representative of 
schools in general. In addition to sampling pro¬ 
cedures, participation rates should be reported, 
and the method of calculating participation 
rates should be clearly described. Studies that are 
designed to be nationally representative often 
use weights so that the weighted sample better 
represents the nation than does the unweighted 
sample. When weights are used, it is important 
that the procedure for deriving the weights be 
described and that the demographic representa- 
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tion of both the weighted and the unweighted 
samples be given. If norming data are collect¬ 
ed under conditions in which student motiva¬ 
tion in completing the test is likely to difFer 
from that expected during operational use, this 
should be clearly documented. Likewise, if the 
instructional histories of students in the norm¬ 
ing sample difFer systematically from those to 
be expected during operational test use, that 
fact should be noted. Norms based on samples 
cannot be perfectly precise. Even though the 
imprecision of norm-referenced interpretations 
due to imperfections in the norms themselves 
may be small compared to that due to meas¬ 
urement error, estimates of the precision of 
norms should be available in technical docu¬ 
mentation. For example, standard errors based 
on the sample design might be presented. In 
some testing applications, norms based on all 
examinees tested over a given period of time 
may be useful for some purposes. Such norms 
should be dearly characterized as being based 
on a sample of persons routinely tested as pair 
of an ongoing testing program. 

Standard 4=7 

If local examinee groups differ materially 
from the populations to which norms refer, a 
user who reports derived scores based on the 
published norms has the responsibility to 
describe such differences if they bear upon 
the interpretation of the reported scores. 

Comment: In employment settings, the qualifi¬ 
cations of local examinee groups may fluctuate 
depending on recruitment or referral proce¬ 
dures as well as market conditions. In such 
cases, appropriate test use and interpretation 
may nor require documentation or cautions 
concerning departures from characteristics of 
the norming population. 

Standard 4.8 

When norms are used to characterize exam¬ 
inee groups, the statistics used to summarize 
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each group’s performance and the norms to 
which those statistics are referred should be 
clearly defined and should support the 
intended use or interpretation. 

Comment: Group means are distributed dif¬ 
ferently from individual scores. For example, 
it is not possible to determine the percentile 
rank of a school’s average test score if all that is 
known are the percentile ranks of each of that 
school’s students. It may sometimes be useful to 
develop special norms for group means, but 
when the sizes of the groups difFer materially 
or when some groups are much more heteroge¬ 
neous than others, the construction and inter¬ 
pretation of group norms is problematical. One 
common and acceptable procedure is to report 
the percentile rank of the median group 
member, for example, the median percentile 
rank of the pupils rested in a given school. 

Standard 4.9 

When raw score or derived score scales are 
designed for criterion-referenced interpreta¬ 
tion, including the classification of exam¬ 
inees into separate categories, the rationale 
for recommended score interpretations 
should be clearly explained. 

Comment: Criterion-referenced interpretations 
are score-based descriptions or inferences that 
do not take the form of comparisons to the test 
performance of other examinees. Examples 
include statements that some psychopathology 
is likely present, that a prospective employee 
possesses specific skills required in a given posi¬ 
tion, or that a child scoring above a certain score 
point can successfully apply a given set of skills. 
Such interpretations may refer to the absolute 
levels of test scores or to patterns of scores for 
an individual examinee. Whenever the test 
developer recommends such interpretations, 
the rationale and empirical basis should be 
clearly presented. Serious efforts should be 
made whenever possible to obtain independent 
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evidence concerning the soundness of such 
score interpretations. Criterion-referenced 
and norm-referenced scales are not mutually 
exclusive. Given adequate supporting data, 
scores may be interpreted by both approaches, 
not necessarily just one or the other. 

Standard 4.10 

A clear rationale and supporting evidence 
should be provided for any claim that scores 
earned on different forms of a test may be 
used interchangeably. In some cases, direct 
evidence of score equivalence may be provid¬ 
ed. In other cases, evidence may come from 
a demonstration that the theoretical assump¬ 
tions underlying procedures for establishing 
score comparability have been sufficiently sat¬ 
isfied. The specific rationale and the evidence 
required will depend in part on the intended 
uses for which score equivalence is claimed. 

Comment: Support should be provided for any 
assertion that scores obtained using different 
items or testing materials, or different testing 
procedures, are interchangeable for some pur¬ 
pose. This standard applies, for example, to 
alternate forms of a paper-and-pencil test or 
to alternate sets of items taken by different 
examinees in computerized adaptive testing. 
It also applies to test forms administered in 
different formats (e.g., paper-and-pencil and 
computerized tests) or test forms designed for 
individual versus group administration. Score 
equivalence is easiest to establish when differ¬ 
ent forms are constructed following identical 
procedures and then equated statistically. When 
that is not possible, for example, in cases where 
different test formats are used, additional evi¬ 
dence may be required to establish the requisite 
degree of score equivalence for the intended 
context and purpose. When recommended 
inferences or actions are based solely on classifi¬ 
cations of examinees into one of two or more 
categories, the rationale and evidence should 
address consistency of classification. If the only 


score repotted and used is a pass-fail decision, 
for example, then the form-to-form equiva¬ 
lence of measurements for examinees far above 
or far below the cut score is of no concern. 
Some testing accommodations may only affect 
the dependence of test scores on capabilities 
irrelevant to the construct the test is intended 
to measure. Use of a large-print edition, for 
example, assures that performance does not 
depend on the ability to perceive standard-size 
print. In such cases, relatively modest studies 
or professional judgment may be sufficient to 
support claims of score equivalence. 

Standard 4.11 

When claims of form-to-form score equiva¬ 
lence are based on equating procedures, 
detailed technical information should be 
provided on the method by which equating 
functions or other linkages were established 
and on the accuracy of equating functions. 

Comment: The fundamental concern is to 
show that equated scores measure essentially 
the same construct, with vety similar levels of 
reliability and conditional standard errors of 
measurement. Technical information should 
include the design of equating studies, che 
statistical methods used, the size and relevant 
characteristics of examinee samples used in 
equating studies, and the characteristics of any 
anchor tests or linking items. Standard errors 
of equating functions should be estimated and 
reported whenever possible. Sample sizes per¬ 
mitting, it may be informative to determine 
equating functions independendy for identifi¬ 
able subgroups of examinees. It may also be 
informative to use two anchor forms and to 
conduct the equadng using each of the anchors. 
In some cases, equadng functions may be deter¬ 
mined independendy using different statistical 
methods. The correspondence of separate func¬ 
tions obtained by such methods can lend sup¬ 
port to the adequacy of the equating results. Any 
substantial disparities found by such methods 
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should be resolved or reported. To be most 
useful, equating error should be presented in 
units of the reported score scale. For testing 
programs with cut scores, equating error near 
the cut score is of primary importance. The 
degree of scrutiny of equating functions should 
be commensurate with the extent of test use 
anticipated and the importance of the deci¬ 
sions the test scores are intended to inform. 

Standard 4.12 

In equating studies that rely on the statisti¬ 
cal equivalence of examinee groups receiving 
different forms, methods of assuring such 
equivalence should be described in detail. 

Comment: Certain equating designs rely on the 
random equivalence of groups receiving different 
forms. Often, one way to assure such equivalence 
is to systematically mix different test forms and 
then distribute them in a random fashion so 
that roughly equal numbers of examinees in 
each group tested receive each form. 

Standard 4.13 

In equating studies that employ an anchor 
test design, the characteristics of the anchor 
test and its similarity to the forms being 
equated should be presented, including both 
content specifications and empirically deter¬ 
mined relationships among test scores. If 
anchor items are used, as in some IRT-based 
and classical equating studies, the represen¬ 
tativeness and psychometric characteristics 
of anchor items should be presented. 

Comment: Tests or test forms may be linked 
via common items embedded within each of 
them, or a common test administered togeth¬ 
er with each of them. These common items 
or tests are referred to as linking items, anchor 
items, or anchor tests. With such methods, 
the quality of the resulting equating depends 
strongly on the adequacy of the anchor tests 


Standard 4.14 

When score conversions or comparison pro¬ 
cedures are used to relate scores on tests or 
test forms that are not closely parallel, the 
construction, intended interpretation, and 
limitations of those conversions or compar¬ 
isons should be clearly described. 

Comment: Various score conversions or con¬ 
cordance tables have been constructed relating 
tests at different levels of difficulty, relating 
earlier to revised forms of published tests, cre¬ 
ating score concordances between different 
tests of similar or different constructs, or for 
other purposes. Such conversions are often 
useful, but they may also be subject to misin¬ 
terpretation. The limitations of such conver¬ 
sions should be clearly described. 

Standard 4.15 

When additional test forms are created by tak¬ 
ing a subset of die items in an existing test form 
or by rearranging its items and there is sound 
reason to believe that scores on these forms 
may be influenced by item context effects, 
evidence should be provided that there is no 
undue distortion of norms for the different 
versions or of score linkages between them. 

Comment: Some tests and test batteries are 
published in both a full-length version and a 
survey or short version. In other cases, multi¬ 
ple versions of a single test form may be cre¬ 
ated by rearranging its items. It should not be 
assumed that performance data derived from 
the administration of items as part of the ini¬ 
tial version can be used to approximate norms 
or construct conversion tables for alternative 
intact tests. Due caution is required in cases 
where contexc effects are likely, including 
speeded tests, long tests where facigue may be 
a factor, and so on. In many cases, adequate 
psychometric data may only be obtainable 
from independent administrations of the 
alternate forms. 
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Standard 4.16 

If test specifications are changed from one 
version of a test to a subsequent version, such 
changes should be identified in the test man¬ 
ual, and an indication should be given that^ 
converted scores for the two versions may not 
be strictly equivalent: When substantial 
changes in test specifications occur, either 
scores should be reported on a new scale or 
a clear statement should be provided to alert 
users that the scores are not direcdy compara¬ 
ble with those on earlier versions of the test. 

Comment: Major shifts sometimes occur in the 
specifications of tests that are used for substan¬ 
tial periods of time. Often such changes take 
advantage of improvements in item types or 
of shifts in content that have been shown to 
improve validity and, therefore, are highly 
desirable. It is important to recognize, howev¬ 
er, chat such shifts will result in scores that 
cannot be made strictly interchangeable with 
scores on an earlier form of the test. 

Standard 4.17 

Testing programs that attempt to maintain 
a common scale over time should conduct 
periodic checks of the stability of the scale 
on which scores are reported. 

Comment: In some testing programs, items are 
introduced into and retired from item pools on 
an ongoing basis. In other cases, the items in suc¬ 
cessive test forms may overlap very little, or not 
at all. In either case, if a fixed scale is used for re¬ 
porting, it is important to assure that the mean¬ 
ing of the scaled scores does not change over time. 

Standard 4.18 

If a publisher provides norms for use in test 
score interpretation, then so long as the test 
remains in print, it is the publishers responsi¬ 
bility to assure that the test is renormed with 
sufficient frequency to permit continued accu¬ 
rate and appropriate score interpretations. 


Comment: Test publishers should assure that 
up-to-date norms are readily available, but it 
remains the test user’s responsibility to avoid 
inappropriate use of norms that are out of date 
and to strive to assure accurate and appropri¬ 
ate test interpretations. 

Standard 4.19 

When proposed score interpretations involve 
one or more cut scores, the rationale and 
procedures used for establishing cut scores 
should be clearly documented. 

Comment: Cut scores may be established to 
select a specified number of examinees (e.g., 
to fill existing vacancies), in which case little 
further documentation may be needed con¬ 
cerning the specific question of how the cut 
scores are established, though attention should 
be paid to legal requirements that may apply. 
In other cases, however, cut scores may be used 
to classify examinees into distinct categories 
(e.g., diagnostic categories, or passing versus 
failing) for which there are no preestablished 
quotas. In these cases, the standard-setting 
method must be clearly documented. Ideally, 
the role of cut scores in test use and interpre¬ 
tation is taken into account during test design. 
Adequate precision in regions of score scales 
where cut points are established is prerequisite 
to reliable classification of examinees into cat¬ 
egories. If standard setting employs data on the 
score distributions for criterion groups or on 
the relation of test scores to one or more criteri¬ 
on variables, those data should be summarized 
in technical documentation. If a judgmental 
standard-setting process is followed, the method 
employed should be clearly described, and the 
precise nature of the judgments called for should 
be presented, whether those are judgments of 
persons, of item or test performances, or of 
other criterion performances predicted by test 
scores. Documentation should also include the 
selection and qualification of judges, training 
provided, any feedback to judges concerning 
the implications of their provisional judgments, 
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and any opportunities forjudges to confer with 
one another. Where applicable, variability over 
judges should be reported. Whenever feasible, an 
estimate should be provided of the amount of 
variation in cut scores that might be expected if 
the standard-setting procedure were replicated. 

Standard 4.20 

When feasible, cut scores defining categories 
with distinct substantive interpretations 
should be established on the basis of sound 
empirical data concerning the relation of test 
performance to relevant criteria. 

Comment: In employment settings, although 
it is important to establish that test scores are 
related to job performance, the precise rela¬ 
tion of test and criterion may have little bear¬ 
ing on the choice of a cut score. However, in 
contexts where distinct interpretations are 
applied to different score categories, the 
empirical relation of test to criterion assumes 
greater importance. Cut scores used in inter¬ 
preting diagnostic tests may be established on 
the basis of empirically determined score dis¬ 
tributions for criterion groups. With achieve¬ 
ment or proficiency tests, such as those used 
in licensure, suitable criterion groups (e.g., 
successful versus unsuccessful practitioners) 
are often unavailable. Nonetheless, it is highly 
desirable, when appropriate and feasible, to 
investigate the relation between test scores 
and performance in relevant practical settings. 
Note that a carefully designed and imple¬ 
mented procedure based solely on judgments 
of content relevance and item difficulty may 
be preferable to an empirical study with an 
inadequate criterion measure or other defi¬ 
ciencies. Professional judgment is required 
to determine an appropriate standard-setting 
approach (or combination of approaches) in 
any given situation. In general, one would 
not expect to find a sharp difference in levels 
of the criterion variable between those just 


below versus just above the cut score, but evi¬ 
dence should be provided where feasible of a 
relationship between test and criterion per¬ 
formance over a score interval that includes 
or approaches the cut score. 

Standard 4.21 

When cut scores defining pass-fail or profi¬ 
ciency categories are based on direct judg¬ 
ments about the adequacy of item or test 
performances or performance levels, the 
judgmental process should be designed so 
that judges can bring their knowledge and 
experience to bear in a reasonable way. 

Comment: Cut scores are sometimes based on 
judgments about the adequacy of item or test 
performances (e.g., essay responses to a writ¬ 
ing prompt) or performance levels (e.g., the 
level that would characterize a borderline 
examinee). The procedures used to elicit such 
judgments should result in reasonable, defensi¬ 
ble standards that accurately reflect the judges’ 
values and intentions. Reaching such judgments 
may be most straightforward when judges are 
asked to consider kinds of performances with 
which they are familiar and for which they 
have formed clear conceptions of adequacy or 
quality. When the responses elicited by a test 
neither sample nor closely simulate the use of 
tested knowledge or skills in the actual criteri¬ 
on domain, judges are not likely to approach 
the task with such clear understandings. Special 
care must then be taken to assure that judges 
have a sound basis for making the judgments 
requested. Thorough familiarity with descrip¬ 
tions of different proficiency categories, prac¬ 
tice in judging task difficulty with feedback 
on accuracy, the experience of actually taking 
a form of the test, feedback on the failure 
rates entailed by provisional standards, and 
ocher forms of information may be beneficial 
in helping judges to reach sound and princi¬ 
pled decisions. 
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Background 

The usefulness, and interpretability of test 
scores require that a test be administered and 
scored according to the developers instruc¬ 
tions. When directions ro examinees, testing 
conditions, and scoring procedures follow the 
same detailed procedures, the test is said to be 
standardized. Without such standardization, 
the accuracy and comparability of score inter¬ 
pretations would be reduced. For tests designed 
to assess the examinees knowledge, skills, or 
abilities, standardization helps to ensure that 
all examinees have the same opportunity to 
demonstrate their competencies. Maintaining 
test security also helps to ensure that no one 
has an unfair advantage. 

Occasionally, however, situations arise in 
which modifications of standardized procedures 
may be advisable or legally mandated. Persons 
of different backgrounds, ages, or familiarity 
with testing may need nonstandard modes of 
test administration or a more comprehensive 
orientation to the testing process, in order that 
all test takers can come to the same under¬ 
standing of the task. Standardized modes of 
presenting information or of responding may 
not be suitable for specific individuals, such 
as persons with some kinds of disability, or 
persons with limited proficiency in the language 
of the test, so that accommodations may be 
needed (see chapters 9 and 10). Large-scale 
testing programs generally have established 
specific procedures to be used in considering 
and granting accommodations. Some test users 
feel that any accommodation not specifically 
required by law could lead to a charge of 
unfair treatment and discrimination. Although 
accommodations are made wich the intent of 
maintaining score comparability, the extent 
to which that is possible may not be known. 
Comparability of scores may be compromised, 
and the test may then not measure the same 
constructs for all test takers. 


Tests and assessments differ in their degree 
of standardization. In many instances different 
examinees are given not the same test form, but 
equivalent forms that have been shown to yield 
comparable scores. Some assessments permit 
examinees to choose which tasks to perform or 
which pieces of their work are to be evaluated. 
A degree of standardization can be maintained 
by specifying the conditions of the choice and 
the criteria of evaluation of the products. When 
an assessment permits a certain kind of collabo¬ 
ration, the limits of that collaboration can be 
specified. With some assessments, test adminis¬ 
trators may be expected to tailor their instruc¬ 
tions to help assure that all examinees understand 
what is expected of them. In all such cases, the 
goal remains the same: to provide accurate and 
comparable measurement for everyone, and 
unfair advantage to no one. The degree of 
standardization is dictated by chat goal, and 
by the intended use of the test. 

Standardized directions to test takers 
help to ensure that all test takers understand 
the mechanics of test caking. Directions gen¬ 
erally inform test takers how to make their 
responses, what kind of help they may legiti¬ 
mately be given if they do not understand 
the question or task, how they can correct 
inadvertent responses, and the nature of any 
time constraints. General advice is some¬ 
times given about omitting item responses. 
Many tests, including computer-administered 
tests, require special equipment. Practice exer¬ 
cises are often presented in such cases to ensure 
that the test taker understands how to operate 
the equipment. The principle of standardiza¬ 
tion includes orienting test takers to materials 
with which they may not be familiar. Some 
equipment may be provided at the testing site, 
such as shop tools or balances. Opportunity 
for rest takers to practice with the equipment 
will often be appropriate, unless using the 
equipment is the purpose of the test. 
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Tests are sometimes administered by 
computer, with test responses made by key¬ 
board, computer mouse, or similar device. 
Although many rest takers are accustomed 
to computers, some are not and may need 
some brief explanation. Even those test tak¬ 
ers who use computers will need to know 
about some details. Special issues arise in 
managing the testing environment, such as 
the arrangement of illumination so that 
light sources do not reflect on the computer 
screen, possibly interfering with display leg¬ 
ibility. Maintaining a quiet environment 
can be challenging when candidates are test¬ 
ed separately, starting at different times and 
finishing at different times from neighbor¬ 
ing test takers. Those who administer com¬ 
puter-based tests require training in the 
hardware and software used for the test, so 
that they can deal with problems that may 
arise in human-computer interactions. 

Standardized scoring procedures help 
to ensure accurate scoring and reporting, 
which are essential in all circumstances. When 
scoring is done by machine, the accuracy of 
the machine is at issue, including any scoring 
algorithm. When scoring is done by human 
judges, scorers require careful training. Regular 
monitoring can also help to ensure that every 
test protocol is scored according to the same 
standardized criteria and that the criteria do 
not change as the test scorers progress through 
the submitted test responses. 

Test scores, per se, are not readily inter¬ 
preted without other information, such as 
norms or standards, indications of measure¬ 
ment error, and descriptions of test content. 
Just as a temperature of 50° in January is 
warm for Minnesota and cool for Florida, a 
test score of 50 is not meaningful without 
some context. When the scores are to be 
reported to persons who are not technical 
specialists, interpretive material can be pro¬ 
vided that is readily understandable to those 
receiving the report. Often, the test user 


provides an interpretation of the results for 
the test taker, suggesting the limitations of 
the resulcs and the relationship of any reported 
scores to other information. Scores on some 
tests are not designed to be released to test 
takers; only broad test interpretations, or 
dichotomous classifications, such as pass/fail, 
are intended to be reported. 

Interpretations of test results are some¬ 
times prepared by computer systems. Such 
interpretations are generally based on a com¬ 
bination of empirical data and expert judg¬ 
ment and experience. In some professional 
applications of individualized testing, the 
computer-prepared interpretations are com¬ 
municated by a professional, possibly with 
modifications for special circumstances. 
Such test interpretations require validation. 
Consistency with interpretations provided by 
nonalgorithmic approaches is clearly a concern. 

In some large-scale assessments, rhe pri¬ 
mary target of assessment is not the individ¬ 
ual test taker but is a larger unit, such as a 
school district or an industrial plant. Often, 
different test takers are given different sets 
of items, following a carefully balanced matrix 
sampling plan, to broaden the range of infor¬ 
mation that can be obtained in a reasonable 
time period. The results acquire meaning 
when aggregated over many individuals taking 
different samples of items. Such assessments 
may not furnish enough information to sup¬ 
port even minimally valid, reliable scores for 
individuals, as each individual may take only 
an incomplete test. 

Some further issues of administration 
and scoring are discussed in chapter 3, “Test 
Development and Revision." 
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Standard 5.1 

Test administrators should follow carefully 
the standardized procedures for administra¬ 
tion and scoring specified by the test devel¬ 
oper, unless the situation or a test takers 
disability dictates that an exception should 
be made. 

Comment: Specifications regarding instruc¬ 
tions to test takers, time limits, the form of 
item presentation or response, and test mate¬ 
rials or equipment should be strictly observed. 
In general, the same procedures should be 
followed as were used when obtaining the 
data for scaling and norming the test scores. 
A test taker with a disabling condition may 
require special accommodation. Other special 
circumstances may require some flexibility in 
administration. Judgments of the suitability 
of adjustments should be tempered by the 
consideration that departures from standard 
procedures may jeopardize the validity of the 
test score interpretations. 

Standard 5.2 

Modifications or disruptions of standardized 
test administration procedures or scoring 
should be documented. 

Comment: Information about the nature of 
modifications of administration should be 
maintained in secure data files, so that research 
studies or case reviews based on test records 
can take this into account. This includes not 
only special accommodations for particular 
test takers, but also disruptions in the testing 
environment that may affect all test takers in 
the testing session. A researcher may wish to 
use only the records based on standardized 
administration. In other cases, research stud¬ 
ies may depend on such information to form 
groups of respondents. Test users or test spon¬ 
sors should establish policies concerning who 
keeps the files and who may have access to 
the files. Whether the information about 


modifications is reported to users of test data, 
such as admissions officers, depends on dif¬ 
ferent considerations (see chapters 8 and 10). 
If such reports are made, certain cautions may 
be appropriate. 

Standard 5.3 

When formal procedures have been estab¬ 
lished for requesting and receiving accom¬ 
modations, test takers should be informed 
of these procedures in advance of testing. 

Comment: When large-scale testing programs 
have established strict procedures to be fol¬ 
lowed, administrators should not depart from 
these procedures. 

Standard 5.4 

The testing environment should furnish rea- 


Commcnt: Noise, disruption in the testing 
area, extremes of temperature, poor lighting, 
inadequate work space, illegible materials, 
and so forth are among the conditions that 
should be avoided in testing situations. The 
testing site should be readily accessible. 
Testing sessions should be monitored where 
appropriate to assist the test taker when a 
need arises and to maintain proper adminis¬ 
trative procedures. In general, the testing 
conditions should be equivalent to those that 
prevailed when norms and other interpreta¬ 
tive data were obtained. 

Standard 5.5 

Instructions to test takers should clearly 
indicate how to make responses. Instructions 
should also be given in the use of any equip¬ 
ment Likely to be unfamiliar to test takers. 
Opportunity to practice responding should 
be given when equipment is involved, unless 
use of the equipment is being assessed. 
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Comment: When electronic calculators are pro¬ 
vided for use, examinees may need practice in 
using the calculator. Examinees may need 
practice responding with unfamiliar tasks, such 
as a numeric grid, which is sometimes used with 
mathematics performance items. In computer- 
administered tests, the method of responding 
may be unfamiliar to some test takers. Where 
possible, the practice responses should be mon¬ 
itored to ensure that the test taker is making 
acceptable responses. In some performance tests 
that involve tools or equipment, instructions may 
be needed for unfamiliar took, unless accommo¬ 
dating to unfamiliar took is pan of what is being 
assessed. If a test taker k unable to use the equip¬ 
ment or make the responses, it may be appropri¬ 
ate to consider alternative testing modes. 

Standard 5.6 

Reasonable efforts should be made to assure 
the integrity of test scores by eliminating 
opportunities for test takers to attain scores 
by fraudulent means. 

Comment: In large-scale testing programs where 
the results may be viewed as having important 
consequences, efforts to assure score integrity 
should include, when appropriate and practi¬ 
cable, stipulating requirements for identifica¬ 
tion, constructing seating charts, assigning 
test takers to seats, requiring appropriate space 
between seats, and providing continuous 
monitoring of the testing process. Test devel¬ 
opers should design test materials and proce¬ 
dures to minimize the possibility of cheating. 
Test administrators should note and report 
any significant instances of testing irregularity. 

A local change in the date or time of testing 
may offer an opportunity for fraud. In gener¬ 
al, steps should be taken to minimize the pos¬ 
sibility of breaches in test security. In any 
evaluation of work products (e.g., portfolios) 
steps should be taken to ensure that the prod¬ 
uct represents the candidate’s own work, and 
that the amount and kind of assistance pro¬ 
vided should be consistent with the intent of 
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the assessment. Ancillary documentation, 
such as the date when the work was done, 
may be useful. 

Standard 5.7 

Test users have the responsibility of protect¬ 
ing the security of test materials at all times. 

Comment: Those who have test materials 
under their control should, with due consid¬ 
eration of ethical and legal requirements, take 
all steps necessary to assure chat only individ¬ 
uals with a legitimate need for access to test 
materials are able to obtain such access before 
the test administration, and afterwards as 
well, if any part of the test will be reused at a 
later time. Test users must balance test securi¬ 
ty with the rights of all test takers and test 
users. When sensitive test documents are 
challenged, it may be appropriate to employ 
an independent third party, using a closely 
supervised secure procedure to conduct a 
review of the relevant materials. Such secure 
procedures are usually preferable to placing 
tests, manuals, and an examinee’s test respons¬ 
es in the public record. 

Standard 5.8 

Test scoring services should document the 
procedures that were followed to assure 
accuracy of scoring. The frequency of scor¬ 
ing errors should be monitored and reported 
to users of the service on reasonable request. 
Any systematic source of scoring errors 
should be corrected. 

Comment: Clerical and mechanical errors 
should be examined. Scoring errors should 
be minimized and, when they are found, 
steps should be taken promptly co minimize 

Standard 5.9 

When test scoring involves human judgment, 
scoring rubrics should specify criteria for scor- 
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ing. Adherence to established scoring criteria 
should be monitored and checked regularly. 
Monitoring procedures should be documented. 

Comment: Human scorers may be provided 
with scoring rubrics listing acceptable alterna¬ 
tive responses, as well as general criteria. 
Consistency of scoring is often checked by 
rescoring randomly selected test responses 
and by rescoring some responses from earlier 
administrations. Periodic checks of the statis¬ 
tical properties (e.g., means, standard devia¬ 
tions) of scores assigned by individual scorers 
during a scoring session can provide feedback 
for the scorers, helping them to maintain 
scoring standards. Lack of consistent scoring 
may call for retraining or dismissing some scor¬ 
ers or for reexamining the scoring rubrics. 

Standard 5.10 

When test score information is released to 
students, parents, legal representatives, teach¬ 
ers, clients, or the media, those responsible 
for testing programs should provide appro¬ 
priate interpretations. The interpretations 
should describe in simple language what the 
test covers, what scores mean, the precision 
of the scores, common misinterpretations of 
test scores, and how scores will be used. 

Comment: Test users should consult the inter¬ 
pretive material prepared by the test developer 
or publisher and should revise or supplement 
the material as necessary to present the local and 
individual results accurately and clearly. Score 
precision might be depicted by error bands, 
or likely score ranges, showing the standard 
error of measurement. 

Standard 5.11 

When computer-prepared interpretations of 
test response protocols are reported, the 
sources, rationale, and empirical basis for 
these interpretations should be available, 
and their limitations should be described. 


Comment: Whereas computer-prepared inter¬ 
pretations may be based on expert judgment, 
the interpretations are of necessity based 
on accumulated experience and may not be 
able to take into consideration the context of 
the individual’s circumstances. Computer- 
prepared interpretations should be used with 
care in diagnostic settings, because they 
may not take into account other information 
about the individual test taker, such as age, 
gender, education, prior employment, and 
medical history, that provide context for 
test results. 

Standard 5.12 

When group-level information is obtained 
by aggregating the results of partial tests 
taken by individuals, validity and reliability 
should be reported for the level of aggrega¬ 
tion at which results are reported. Scores 
should not be reported for individuals unless 
the validity, comparability, and reliability of 
such scores have been established. 

Comment: Large-scale assessments often 
achieve efficiency by “matrix sampling” of 
the content domain by asking different test 
takers different questions. The testing then 
requires less time from each test taker, while 
the aggregation of individual results provides 
for domain coverage that can be adequate 
for meaningful group- or program-level 
interpretations, such as schools, or grade 
levels within a locality or particular subject- 
matter areas. Because the individual receives 
only an incomplete test, an individual score 
would have limited meaning. If individual 
scores are provided, comparisons between 
scores obtained by different individuals are 
based on responses to items that may cover 
different material. Some degree of calibra¬ 
tion among incomplete tests can sometimes 
be made. Such calibration is essential to the 
comparisons of individual scores. 
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Standard 5.13 

Transmission of individually identified test 
scores to authorized individuals or institu¬ 
tions should be done in a manner that pro¬ 
tects the confidential nature of the scores. 

Comment: Care is always needed when com¬ 
municating the scores of identified test takers, 
regardless of the form of communication. 
Face-to-face communication, as well as tele¬ 
phone and written communication present 
well-known problems. Transmission by elec¬ 
tronic media, including computer networks 
and facsimile, presents modern challenges 
to confidentiality. 

Standard 5.14 

When a material error is found in test scores 
or other important information released by a 
testing organization or other institution, a 
corrected score report should be distributed 
as soon as practicable to all known recipients 
who might otherwise use the erroneous scores 
as a basis for decision making. The corrected 
report should be labeled as such. 

Comment: A material error is one that could 
change the interpretation of the test score. 
Innocuous typographical errors would be 
excluded. Timeliness is essential for decisions 
that will be made soon after the test scores 
are received. 

Standard 5.15 

When test data about a person are retained, 
both the test protocol and any written 
report should also be preserved in some 
form. Test users should adhere to the poli¬ 
cies and record-keeping practice of their 
professional organizations. 

Comment: The protocol may be needed to 
respond ro a possible challenge from a test 
taker. The protocol would ordinarily be 


accompanied by testing materials and test 
scores. Retention of more detailed records of 
responses would depend on circumstances 
and should be covered in a retention policy 
(see the following standard). Record keeping 
may be subject to legal and professional 
requirements. Policy for the release of any test 
information for other than research purposes 
is discussed in chapter 8. 

Standard 5.16 

Organizations that maintain test scores on 
individuals in data files or in an individual’s 
records should develop a clear set of policy 
guidelines on the duration of retention of an 
individual’s records, and on the availability, 
and use over time, of such data. 

Comment: In some instances, test scores 
become obsolete over time, no longer 
reflecting the current state of the test taker. 
Outdated scores should generally not be used 
or made available, except for research purpos¬ 
es. In other cases, test scores obtained in past 
years can be useful as, for example, in longi¬ 
tudinal assessment. The key issue is the valid 
use of the information. Score retention and 
disclosure may be subject to legal and profes¬ 
sional requirements. 
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Background 

The provision of supporting documents for 
tests is the primary means by which test 
developers, publishers, and distributors com¬ 
municate with test users. These documents 
are evaluated on the basis of their complete¬ 
ness, accuracy, currency, and clarity and 
should be available to qualified individuals as 
appropriate. A test’s documentation typically 
specifies the nature of the test; its intended 
use; the processes involved in the test’s devel¬ 
opment; technical information related to 
scoring, interpretation, and evidence of valid¬ 
ity and reliability; scaling and norming if 
appropriate to the instrument; and guidelines 
for test administration and interpretation. 
The objective of the documentation is to pro¬ 
vide test users with the information needed to 
make sound judgments about the nature and 
quality of the test, the resulting scores, and 
the interpretations based on the test scores. 
The information may be reported in docu¬ 
ments such as test manuals, technical manu¬ 
als, user’s guides, specimen sets, examination 
kits, directions for test administrators and 
scorers, or preview materials for test takers. 

Test documentation is most effective if it 
communicates information to multiple user 
groups. To accommodate the breadth of 
training of professionals who use tests, sepa¬ 
rate documents or sections of documents may 
be written for identifiable categories of users 
such as practitioners, consultants, administra¬ 
tors, researchers, and educators. For example, 
the test user who administers the tests and 
interprets the results needs interpretive infor¬ 
mation or guidelines. On the other hand, 
those who are responsible for selecting tests 
need to be able to judge the technical adequa¬ 
cy of the test. Therefore, some combination 
of technical manuals, user’s guides, test man¬ 
uals, test supplements, examination kits, or 


specimen sets ordinarily is published to pro¬ 
vide a potential test user or test reviewer with 
sufficient information to evaluate the appro¬ 
priateness and technical adequacy of the test. 
The types of information presented in these 
documents typically include a description of 
the intended test-taking population, stated 
purpose of the test, test specifications, item 
formats, scoring procedures, and the test 
development process. Technical data, such as 
psychometric indices of the items, reliability 
and validity evidence, normative data, and 
cut scores or configural rules including those 
for computer-generated interpretations of test 
scores also are summarized. 

An essential feature of the documentation 
for every test is a discussion of the known 
appropriate and inappropriate uses and inter¬ 
pretations of the test scores. The inclusion of 
illustrations of score interpretations, as they 
relate to the test developer’s intended applica¬ 
tions, also will help users make accurate infer¬ 
ences on the basis of the test scores. When 
possible, illustrations of improper test uses and 
inappropriate test score interpretations will 
help guard against the misuse of the test. 

Test documents need to include enough 
information to allow test users and reviewers 
to determine the appropriateness of the test 
for its intended purposes. References to other 
materials that provide more details about 
research by the publisher or independent 
investigators should be cited and should be 
readily obtainable by the test user or reviewer. 
This supplemental material can be provided 
in any of a variety of published or unpub¬ 
lished forms; when demand is likely to be 
low, it may be maintained in archival form, 
including electronic storage. Test documenta¬ 
tion is useful for all test instruments, includ¬ 
ing those that are developed exclusively for 
use within a single organization. 
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In addition to technical documentation, 
descriptive materials are needed in some set¬ 
tings to inform examinees and other interested 
parties about the nature and content of the 
test. The amount and type of information 
will depend on the particular test and appli¬ 
cation. For example, in situations requiring 
informed consent, information should be suf¬ 
ficient to develop a reasoned judgment. Such 
information should be phrased in nontechni¬ 
cal language and should be as inclusive as is 
consistent with the use of the test scores. The 
materials may include a general description 
and rationale for the test; sample items or 
complete sample tests; and information about 
conditions of test administration, confiden¬ 
tiality, and retention of test results. For some 
applications, however, the true nature and 
purpose of a test are purposely hidden or dis¬ 
guised to prevent faking or response bias. In 
these instances, examinees may be motivated 
to reveal more or less of the characteristics 
intended to be assessed. Under these circum¬ 
stances, hiding or disguising the true nature 
or purpose of the rest is acceptable provided 
this action is consistent with legal principles 
and ethical standards. 

This chapter provides general standards 
for the preparation and publication of test 
documentation. The other chapters contain 
specific standards that will be useful to test 
developers, publishers, and distributors in the 
preparation of materials to be included in a 
test’s documentation. 


Standard 6.1 

Test documents (e.g., test manuals, technical 
manuals, user’s guides, and supplemental 
material) should be made available to prospec¬ 
tive test users and other qualified persons at 
the time a test is published or released for use. 

Comment: The test developer or publisher 
should judge carefully which information 
should be included in first editions of the test 
manual, technical manual, or user’s guides 
and which information can be provided in 
supplements. For low-volume, unpublished 
tests, the documentation may be relatively brief. 
When the developer is also the user, docu¬ 
mentation and summaries are still necessary. 

Standard 6.2 

Test documents should be complete, accu¬ 
rate, and dearly written so that the intended 
reader can readily understand the content. 

Comment: Test documents should provide 
sufficient detail to permit reviewers and 
researchers to judge or replicate important 
analyses published in the test manual. For 
example, reporting correlation matrices in 
the test document may allow the test user 
to judge the data upon which decisions and 
conclusions wete based, or describing in 
detail the sample and the nature of any factor 
analyses that were conducted will allow the 
test user to replicate reported studies. 

Standard 6.3 

The rationale for the test, recommended 
uses of the test, support for such uses, and 
information that assists in score interpreta¬ 
tion should be documented. Where particu¬ 
lar misuses of a test can be reasonably 
anticipated, cautions against such misuses 
should be specified. 

Comment: Test publishers make every effort 
to caution test users against known misuses of 
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STANDARDSl 


tests. However, test publishers are not required 
to anticipate all possible misuses of a test. If 
publishers do know of persistent test misuse 
by a test user, extraordinary educational 
efforts may be appropriate. 

Standard 6.4 

The population for whom the test is intended 
and the test specifications should be docu¬ 
mented. If applicable, the item pool and scale 
development procedures should be described 
in the relevant test manuals. If normative data 
are provided, the norming population should 
be described in terms of relevant demographic 
variables, and the year(s) in which the data 
were collected should be reported. 

Comment: Known limitations of a test for cer¬ 
tain populations also should be clearly delin¬ 
eated in the test documents. In addition, if 
the test is available in more than one language, 
test documents should provide information 
on the translation or adaptation procedures, 
on the demographics of each norming sample, 
and on score interpretation issues for each lan¬ 
guage into which the test has been translated. 

Standard 6.5 

When statistical descriptions and analyses 
that provide evidence of the reliability of 
scores and the validity of their recommended 
interpretations are available, the information 
should be included in the test’s documenta¬ 
tion. When relevant for test interpretation, 
test documents ordinarily should include 
item level information, cut scores and con- 
figurai rules, information about raw scores 
and derived scores, normative data, the stan¬ 
dard errors of measurement, and a descrip¬ 
tion of the procedures used to equate 
multiple forms. 

Standard 6.6 

When a test relates to a course of training or 
study, a curriculum, a textbook, or packaged 


instruction, the documentadon should include 
an identification and description of the course 
or instructional materials and should indicate 
the year in which these materials were prepared. 

Standard 6.7 

Test documents should specify qualifications 
that are required to administer a test and to 
interpret the test scores accurately. 

Comment: Statements of user qualifications 
need to specify the training, certification, 
competencies, or experience needed to have 
access to a test. 

Standard 6.8 

If a test is designed to be scored or interpre¬ 
ted by test takers, the publisher and test 
developer should provide evidence that the 
test can be accurately scored or interpreted 
by the test takers. Tests that are designed to 
be scored and interpreted by the test taker 
should be accompanied by interpretive 
materials that assist the individual in under¬ 
standing the test scores and that are written 
in language that the test taker can understand. 

Standard 6.9 

Test documents should cite a representative 
set of the available studies pertaining to gen¬ 
eral and specific uses of the test. 

Comment: Summaries of cited studies—exclud¬ 
ing published works, dissertations, or propri¬ 
etary documents—should be made available 
on request to test users and researchers by the 
publisher. 

Standard 6.10 

Interpretive materials for tests, that include 
case studies, should provide examples illus¬ 
trating the diversity of prospective test takers. 

Comment: For some instruments, the presen¬ 
tation of case studies that are intended to 
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assist the user in the interpretation of the test 
scores and profiles also will be appropriate for 
inclusion in the test documentation. For 
example, case studies might cite as appropri¬ 
ate examples of women and men of different 
ages; individuals differing in sexual orienta¬ 
tion; persons representing various ethnic, cul¬ 
tural, or racial groups; and individuals with 
special needs. The inclusion of examples illus¬ 
trating the diversity of prospective test takers 
is not intended to promote interpretation of 
test scores in a manner inconsistent with legal 
requirements that may restrict certain practices 
in some contexts, such as employee selection. 

Standard 6.11 

If a test is designed so that more than one 
method can be used for administration or 
for recording responses—such as marking 
responses in a test booklet, on a separate 
answer sheet, or on a computer keyboard— 
then the manual should clearly document the 
extent to which scores arising from these 
methods are interchangeable. If the results 
are not interchangeable, this fact should be 
reported, and guidance should be given for 
the interpretation of scores obtained under 
the various conditions or methods of 
administration. 

Standard 6.12 

Publishers and scoring services that offer 
computer-generated interpretations of test 
scores should provide a summary of the evi¬ 
dence supporting the interpretations given. 

Comment: The test user should be informed 
of any cut scores or configural rules necessary 
for understanding computer-generated score 
interpretations. A description of both the sam¬ 
ples used to derive cut scores or configural rules 
and the methods used to derive the cut scores 
should be provided. When proprietary inter¬ 
ests result in the withholding of cut scores or 
configural rules, the owners of the intellectual 


property are responsible for documenting evi¬ 
dence in support of the validity of computer- 
generated score interpretations. Such evidence 
might be provided, for example, by reporting 
the finding of an independent review of the 
algorithms by qualified professionals. 

Standard 6.13 

When substantial changes are made to a 
test, the test’s documentation should be 
amended, supplemented, or revised to keep 
information for users current and to provide 
useful additional information or cautions. 

Standard 6.14 

Every test form and supporting document 
should carry a copyright date or publication 
date. 

Comment: During the operational life of a test, 
new or revised test forms may be published, 
and manuals and other materials may be 
added or revised. Users and potential users 
are entitled to know the publication dates of 
various documents that include test norms. 
Communication among researchers is ham¬ 
pered when the particular test documents 
used in experimental studies are ambiguously 
referenced in research reports. 

Standard 6.15 

Test developers, publishers, and distributors 
should provide general information for test 
users and researchers who may be required 
to determine the appropriateness of an 
intended test use in a specific context. When 
a particular test use cannot be justified, the 
response to an inquiry from a prospective test 
user should indicate this fact clearly. General 
information also should be provided for test 
takers and legal guardians who must provide 
consent prior to a test’s administration. 
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7. FAIRNESS IN TESTING AND 
TEST USE 


Background 

This chapter addresses overriding issues of 
fairness in testing. It is intended both to 
emphasize the importance of fairness in all 
aspects of testing and assessment and to serve 
as a context for the technical standards. Later 
chapters address in greater detail some fairness 
issues involving the responsibilities of test 
users, the rights and responsibilities of test 
takers, the testing of individuals of diverse lin¬ 
guistic backgrounds, and the testing of those 
with disabilities. Chapters 12 through 15 also 
address some fairness issues specific to psycho¬ 
logical, educational, employment and creden- 
tialing, and program evaluation applications 
of testing and assessment. 

Concern for fairness in testing is perva¬ 
sive, and the treatment accorded the topic 
here cannot do justice to the complex issues 
involved. A full consideration of fairness 
would explore the many functions of testing 
in relation to its many goals, including the 
broad goal of achieving equality of opportu¬ 
nity in our society. It would consider the 
technical properties of tests, the ways test 
results are reported, and the factors that are 
validly or erroneously thought to account 
for patterns of test performance for groups 
and individuals. A comprehensive analysis 
would also examine the regulations, statutes, 
and case law that govern test use and the 
remedies for harmful practices. The Standards 
cannot hope to deal adequately with all these 
broad issues, some of which have occasioned 
sharp disagreement among specialists and 
other thoughtful observers. Rather, the focus 
of the Standards is on those aspects of tests, 
testing, and test use that are the customary 
responsibilities of those who make, use, 
and interpret tests, and that are character¬ 
ized by some measure of professional and 
technical consensus. 


Absolute fairness to every examinee is 
impossible to attain, if for no other reasons 
than the facts that tests have imperfect relia¬ 
bility and that validity in any particular con¬ 
text is a matter of degree. But neither is any 
alternative selection or evaluation mechanism 
perfectly fair. Properly designed and used, 
tests can and do further societal goals of fair¬ 
ness and equality of opportunity. Serious 
technical deficiencies in test design, use, or 
interpretation should, of course, be addressed, 
but the fairness of testing in any given con¬ 
text must be judged relative to that of feasible 
test and nontest alternatives. It is general 
practice that large-scale tests are subjected to 
careful review and empirical checks to mini¬ 
mize bias. The amount of explicit attention to 
fairness in the design of well-made tests com¬ 
pares favorably to that of many alternative 
selection or evaluation methods. 

It is also crucial to bear in mind that test 
settings are interpersonal. The interaction of 
examiner with examinee should be profes¬ 
sional, courteous, caring, and respectful. In 
most testing situations, the roles of examiner 
and examinee are sharply unequal in status. A 
professionals inferences and reports from test 
findings may markedly impact the life of the 
person who is examined. Attention to these 
aspects of test use and interpretation is no less 
important than more technical concerns. 

As is emphasized in professional educa¬ 
tion and training, users of tests should be 
alert to the possibility that human issues 
involving examiner and examinee may some¬ 
times affect test fairness. Attention to inter¬ 
personal issues is always important, perhaps 
especially so when examinees have a disability 
or differ from the examiner in ethnic, racial, 
or religious background; in gender or sexual 
orientation; in socioeconomic status; in age; 
or in other respects that may affect the exam¬ 
inee-examiner interaction. 
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Varying Views of Fairness 

The term fairness is used in many different ways 
and has no single technical meaning. It is pos¬ 
sible that two individuals may endorse fairness 
in testing as a desirable social goal, yet reach 
quite different conclusions about the fairness 
of a given testing program. Outlined below are 
four principal ways in which the term fairness 
is used. It should be noted, however, that 
many additional interpretations may be found 
in the technical and popular literature. 

The first two characterizations presented 
here relate fairness to absence of bias and to 
equitable treatment of all examinees in the 
testing process. There is broad consensus that 
tests should be free from bias (as defined 
below) and that all examinees should be treat¬ 
ed fairly in the testing process itself (e.g., 
afforded the same or comparable procedures in 
testing, test scoring, and use of scores). The 
third characterization of test fairness addresses 
the equality of testing outcomes for examinee 
subgroups defined by race, ethnicity, gender, 
disability, or other characteristics. The idea that 
fairness requires equality in overall passing 
rates for different groups has been almost 
entirely repudiated in the professional tesdng 
literature. A more widely accepced view would 
hold that examinees of equal standing with 
respect to the construct the test is intended to 
measure should on average earn the same test 
score, irrespective of group membership. 
Unfortunately, because examinees’ levels of 
the construct are measured imperfectly, this 
requirement is rarely amenable to direct exami¬ 
nation. The fourth definition of fairness relates 
to equity in opportunity to learn the material 
covered in an achievement test. There would 
be general agreement that adequate opportuni¬ 
ty to learn is clearly relevant to some uses and 
interpretations of achievement tests and clearly 
irrelevant to others, although disagreement might 
arise as to the relevance of opportunity to learn 
to cest fairness in some specific situations. 


Fairness as Lack of Bias 

Bias is used here as a technical term. It is 
said to arise when deficiencies in a test itself 
or the manner in which it is used result in 
different meanings for scores earned by mem¬ 
bers of different identifiable subgroups. When 
evidence of such deficiencies is found at the 
level of icem response patterns for members 
of different groups, the terms item bias or dif¬ 
ferential item functioning (DIF) are often used. 
When evidence is found by comparing the 
patterns of association for different groups 
between test scores and other variables, the 
term predictive bias may be used. The concept 
of bias and techniques for its detection are 
discussed below and are also discussed in 
other chapters of the Standards. There is 
general consensus that consideration of bias 
is critical to sound testing practice. 

Fairness as Equitable Treatment in the Testing 
Process 

There is consensus chat just treatment 
throughout the testing process is a necessary 
condition for test fairness. There is also con¬ 
sensus that fair treatment of all examinees 
requires consideration not only of a test itself, 
but also the context and purpose of testing 
and the manner in which test scores are used. 
A well-designed cest is not intrinsically fair or 
unfair, but the use of the cesc in a particular 
circumstance or with particular examinees 
may be fair or unfair. Unfairness can have 
individual and collective consequences. 

Regardless of the purpose of testing, fair¬ 
ness requires that all examinees be given a 
comparable opportunity to demonstrate 
their standing on the construct(s) the cest is 
intended to measure. Just treatment also 
includes such factors as appropriate testing 
conditions and equal opportunity to become 
familiar with the test format, practice materi¬ 
als, and so forth. In situations where individ¬ 
ual or group cest results ate reported, just 
treatment also implies that such reporting 
should be accurate and fully informative. 
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Fairness also requires that a!! examinees 
be afforded appropriate testing conditions. 
Careful standardization of tests and admin¬ 
istration conditions generally helps to assure 
that examinees have comparable opportuni¬ 
ty to demonstrate the abilities Or attributes 
to be measured. In some cases, however, 
aspects of the testing process that pose no 
particular challenge for most examinees may 
prevent specific groups or individuals from 
accurately demonstrating their standing 
with respect to the construct of interest 
(e.g., due to disability or language back¬ 
ground). In some instances, greater compa¬ 
rability may sometimes be attained if 
standardized procedures are modified. There 
are contexts in which some such modifica¬ 
tions are forbidden by law and other con¬ 
texts in which some such modifications are 
required by law. In all cases, standardized 
procedures should be followed for all exam¬ 
inees unless explicit, documented accommo¬ 
dations have been made. 

Ideally, examinees would also be afford¬ 
ed equal opportunity to prepare for a test. 
Examinees should in any case be afforded 
equal access to materials provided by the 
testing organization and sponsor which 
describe the test content and purpose and 
offer specific familiarization and preparation 
for test taking. In addition to assuring equi¬ 
ty in access to accepted resources for rest 
preparation, this principle covers test securi¬ 
ty for nondisclosed tests. If some examinees 
were to have prior access to the contents of 
a secure rest, for example, basing decisions 
upon the relative performance of different 
examinees would be unfair to others who 
did not have such access. On tescs that have 
important individual consequences, all exam¬ 
inees should have a meaningful opportunity 
to provide input to relevant decision makers 
if procedural irregularities in testing are 
alleged, if the validity of the individuals 
score is challenged or may not be reported, 
or if similar special circumstances arise. 


Finally, the conception of fairness as 
equitable treatment in the testing process 
extends to the reporting of individual and 
group test results. Individual test score infor¬ 
mation is entided to confidential treatment in 
most Circumstances: Confidential ity should 
be respected; scores should be disclosed only 
as appropriate. When test scores are reported, 
either for groups or individuals, score reports 
should be accurate and informative. It may 
be especially important when reporting 
results to nonprofessional audiences to use 
appropriate language and wording and to 
try to design reports to reduce the likelihood 
of inappropriate interpretations. When group 
achievement differences are reported, for 
example, including additional information to 
help the intended audience understand con¬ 
founding factors such as unequal educational 
opportunity may help to reduce misinterpre¬ 
tation of test results and increase the likeli¬ 
hood that tests will be used wisely. 

Fairness as Equauty in Outcomes of Testing 
The idea that fairness requires overall 
passing rates to be comparable across groups 
is not generally accepted in the professional 
literature. Most testing professionals would 
probably agree that while group differences in 
testing outcomes should in many cases trigger 
heightened scrutiny for possible sources of 
test bias, outcome differences across groups 
do not in themselves indicate that a testing 
application is biased or unfair. It might be 
argued that when tests are used for selection, 
persons who all would perform equally well 
on the criterion measure if selected should 
have an equal chance of being chosen regard¬ 
less of group membership. Unfortunately, 
there is rarely any direct procedure for deter¬ 
mining whether this ideal has been met. 
Moreover, if score distributions differ from 
one group to another, it is generally impossi¬ 
ble to satisfy this ideal using any test that has 
a less than perfect correlation with the criteri¬ 
on measure. 
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Many testing professionals would agree 
that if a test is free of bias and examinees 
have received fair treatment in the testing 
process, then the conditions of fairness have 
been met. That is, given evidence of the 
validity of intended test uses and interpreta¬ 
tions, including evidence of lack of bias and 
attention to issues of fair treatment, fairness 
has been established regardless of group-level 
outcomes. This view need not imply that 
unequal testing outcomes should be ignored 
altogether. They may be important in gener¬ 
ating new hypotheses about bias and fair 
treatment. But in this view, unequal out¬ 
comes at the group level have no direct bear¬ 
ing on questions of test fairness. There may 
be legal requirements to investigate certain 
differences in outcomes of testing among sub¬ 
groups. Those requirements further may pro¬ 
vide that, other things being equal, a testing 
alternative that minimizes outcome differ¬ 
ences across relevant subgroups should be 
used. The standards in this chapter are 
intended to be applied in a manner consistent 
with legal and regulatory standards. 

Fairness as Opportunity to Learn 

This final conception of fairness arises in 
connection with educational achievement test¬ 
ing. In many contexts, achievement tests are 
intended to assess what a test taker knows ot 
can do as a result of formal instruction. When 
some test takers have not had the opportunity 
to learn the subject matter covered by the test 
content, they are likely to get low scores. The 
test score may accurately reflect what the test 
taker knows and can do, but low scores may 
have resulted in part from not having had the 
opportunity to learn the material tested as well 
as from having had the opportunity and having 
failed to learn. When test takers have not had 
the opportunity to learn the material tested, the 
policy of using cheir test scores as a basis for 
withholding a high school diploma, for exam¬ 
ple, is viewed as unfair. This issue is further dis¬ 
cussed in chapter 13, on educational testing. 
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At ieast three important difficulties arise 
with this conception of fairness. First, the 
definition of opportunity to learn is difficult in 
practice, especially at the level of individuals. 
Opportunity is a matter of degree. Moreover, 
the measurement of some important learning 
outcomes may require students to work with 
material they have not seen before. Second, 
even if it is possible to document the topics 
included in the curriculum for a group of stu¬ 
dents, specific content coverage for any one 
student may be impossible to determine. 
Finally, there is a well-founded desire to 
assure that credentials attest to certain profi¬ 
ciencies or capabilities. Granting a diploma to 
a low-scoring examinee on the grounds that 
the student had insufficient opportunity to 
learn the material tested means certificating 
someone who has not attained the degree of 
proficiency the diploma is intended to signify. 

It should be noted that opportunity to 
learn ordinarily plays no role in determining 
che fairness of tests used for employment and 
credentialing, which are covered in chapter 
14, nor of admissions testing. In those cir¬ 
cumstances, it is deemed fair that the test 
should cover the full range of requisite 
knowledge and skills. However, there are situ¬ 
ations in which the agency that determines 
the contents of a test used for employment or 
credentialing also sets the curriculum that 
must be followed in preparing to take the 
test. In such cases, it is the responsibility of 
that agency to assure that what is to be tested 
is fully included in the specification of what 
is to be taught. 

Bias Associated With Test Content 
and Response Processes 

The term bias in tests and testing refers to 
construct-irrelevant components that result 
in systematically lower or higher scores for 
identifiable groups of examinees. Such con¬ 
struct-irrelevant score components may be 
introduced due to inappropriate sampling of 
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test content or lack of clarity in test instruc¬ 
tions. They may also arise if scoring criteria 
fail to credit fully some correct problem 
approaches or solutions that are more typi¬ 
cal of one group than another. Evidence of 
these potential sources of bias may be 
sought in the content of the tests, in com¬ 
parisons of the internal structure of test 
responses for different groups, and in com¬ 
parisons of the relationships of test scores 
to other measures, although none of these 
types of evidence is unequivocal. 

Content-Related Sources of Test Bias 

Bias due to inappropriate selection of 
test content may sometimes be detected by 
inspection of the test itself. In some testing 
contexts, it is common for test developers to 
engage an independent panel of diverse 
experts to review test content for language 
that might be interpreted differently by mem¬ 
bers of different groups and for material that 
might be offensive or emotionally disturbing 
to some test takers. For performance assess¬ 
ments, panels are often engaged to review 
the scoring rubric as well. A test intended to 
measure verbal analogical reasoning, for 
example, should include words in general use, 
not words and expressions associated with 
particular disciplines, occupations, ethnic 
groups, or locations. Where material likely 
to be differentially interesting or relevant to 
some examinees is included, it may be bal¬ 
anced by material that may be of particular 
interest to the remaining examinees. 

In educational achievement testing, 
alignment with curriculum may bear on ques¬ 
tions of content-related test bias. One may 
ask how well a test represents some content 
domain and also whether that domain is 
appropriate given intended score interpreta¬ 
tions. A test of 19th-century United States 
history might give considerable emphasis to 
the War of 1812, the Mexican War, the Civil 
War, and the Spanish American War. If some 
states curriculum framework dealt relatively 


lighdy with these wars, devoting more atten¬ 
tion instead, say, to social and industrial 
developments, then that states test takers 
might be relatively disadvantaged. 

Bias may also result from a lack of clarity 
in test instructions or from scoring rubrics 
that credit responses more typical of one 
group than another. For example, cognitive 
ability tests often require test takers to classify 
objects according to an unspecified rule. If a 
given task credits classification on the basis of 
the stimulus objects’ functions, but an identi¬ 
fiable subgroup of examinees tends to classify 
the objects on the basis of their physical 
appearance, faulty test interpretations are 
likely. Similarly, if the scoring rubric for a 
constructed response item reserves the highest 
score level for those examinees who in fact 
provide more information or elaboration than 
was actually requested, then less test-wise 
examinees who simply follow instructions will 
earn lower scores. In this case, testwiseness 
becomes a construct-irrelevant component 
of test scores. 

Judgmental methods for the review of 
tests and test items are often supplemented by 
statistical procedures for identifying items on 
tests that function differently across identifi¬ 
able subgroups of examinees. Differential 
item functioning (DIF) is said to exist when 
examinees of equal ability differ on average, 
according to their group membership, in their 
responses to a particular item. If examinees 
from each group are divided into subgroups 
according to the tested ability and subgroups 
at the same ability level have unequal proba¬ 
bilities of answering a given item correctly, 
then there is evidence that that item may not 
be functioning as intended. It may be meas¬ 
uring something different from the remainder 
of the test or it may be measuring with differ¬ 
ent levels of precision for different subgroups 
of examinees. Such an item may offer a valid 
measurement of some narrow element of the 
intended construct, or it may tap some con¬ 
struct-irrelevant component that advantages 
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or disadvantages members of one group. 
Although DIF procedures may hold some 
promise for improving test quality, there has 
been little progress in identifying the causes 
or substantive themes that characterize items 
exhibiting DIF. That is, once items on a test 
have been statistically identified as function¬ 
ing differently from one examinee group to 
another, it has been difficult to specify the 
reasons for the differential performance ot 
to identify a common deficiency among the 
identified items. 

Response-Related Sources of Test Bias 

In some cases, construct-irrelevant score 
components may arise because test items elic¬ 
it varieties of responses other than those 
intended or can be solved in ways that were 
not intended. For example, clients responding 
to a diagnostic inventory may attempt to pro¬ 
vide the answers they think the test adminis¬ 
trator expects as opposed to the answers that 
best describe themselves. To the extent that 
such response acquiescence is more typical 
of some groups than others, bias may result. 
Bias may also be associated with test response 
formats that pose particular difficulties for 
one group or another. For example, test per¬ 
formance may rely on some capability (e.g., 
English language proficiency or fine-motor 
coordination) that is irrelevant to the intent 
of the measurement but nonetheless poses 
impediments for some examinees. A test of 
quantitative reasoning that makes inappropri¬ 
ately heavy demands on verbal ability would 
probably be biased against examinees whose 
first language is other than that of the test. 

In addition to content reviews and DIF 
analyses, evidence of bias related to response 
processes may be provided by comparisons of 
the internal structure of the test responses for 
different groups of examinees. If an analysis 
of the factors or dimensions underlying test 
performance reveals different internal struc¬ 
tures for different groups, it may be that dif¬ 
ferent constructs are being measured or it 


may simply be that groups differ in their vari¬ 
ability with respect to the same underlying 
dimensions. When there is evidence that 
tests, including personality tests, measure dif¬ 
ferent constructs in different gender, racial, or 
cultural groups, it is important to determine 
thac the internal structure of the test supports 
inferences made for clients from these distinct 
subgroups of the client population. In situa¬ 
tions where internal test structure varies 
markedly across ethnically diverse cultures, it 
may be inappropriate to make direct compar¬ 
isons of scores of members of these different 
cultural groups. 

Bias may also be indicated by patterns 
of association between rest scores and other 
variables. Perhaps the most familiar form 
such evidence may take is a difference across 
groups in the regression equations relating 
selection test performance to criterion per¬ 
formance. This case is discussed at greater 
length in the following section. However, 
evidence of bias based on relations to other 
variables may also take many other forms. 
The relationship between two tests of the 
same cognitive ability might be found to dif¬ 
fer from one group to another, for example. 
Such a difference might indicate bias in one 
or both tests. As another instance, a higher 
than expected association between reading 
and mathematics achievement test scores 
among students who might well have limit¬ 
ed English proficiency could trigger an 
investigation to determine whether language 
proficiency was influencing some examinees’ 
mathematics scores. Patterns of score aver¬ 
ages ot other distributional summaries might 
also point to potential sources of test bias. If 
males outperformed females on one measure 
of academic performance and, in the same 
population, females outperformed males on 
another, it would follow that the two meas¬ 
ures could not both be linearly related to the 
identical underlying construct. Note, howev¬ 
er, that if the tested populations differed, if 
the content domains sampled differed, or if 


78 


JA2320 


AERA_APA_NCME_0000087 


Case l:14-cv-00857-TSC Document 60-25 Filed 12/21/15 Page 89 of 100 
USCA Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 562 of 573 

PART II / FAIRNESS IN TESTING AND TEST USE 


the constructs tested otherwise differed due 
to varying motivational contexts or other 
effects, two reliable tests, each valid for its 
intended purpose, might show such a pat¬ 
tern. Association need not imply any direct 
or causal linkage, and alternative explana¬ 
tions for patterns of association should 
usually be considered. In some cases, a test- 
criterion correlation may arise because the 
test and criterion both depend on the same 
construct-irrelevant ability. If identifiable 
subgroups differ with respect to that extra¬ 
neous ability, then bias may result. 

Fairness in Selection and 
Prediction 

When tests are used for selection and predic¬ 
tion, evidence of bias or lack of bias is gener¬ 
ally sought in the relationships between test 
and criterion scores for the respective groups. 
Under one broadly accepred definition, no 
bias exists if the regression equations relating 
the test and the criterion are indistinguishable 
for the groups in question. (Some formula¬ 
tions may hold that not only regression slopes 
and intercepts but also standard errors of 
estimate must be equal.) If test-criterion 
relationships differ, different decision rules 
may be followed depending on the group 
to which the person belongs. 

If fitting a common prediction equation 
for all groups combined suggests that the cri¬ 
terion performance of persons in any one 
group is systematically overpredicted or 
underpredicted, and if bias in the criterion 
measure has been set aside as a possible 
explanation, one possibility is to generate a 
separate prediction formula for each group. 
Another possibility is to seek predictor vari¬ 
ables that may be used in lieu of or in addi¬ 
tion to the initial predictor score to reduce 
differential prediction without reducing over¬ 
all predictive accuracy. If separate regression 
equations are employed, the effect of their 
use on the distribution of predicted criterion 


scores for the different groups should be 
examined. Note that in the United States, the 
use of different selection rules for identifiable 
subgroups of examinees is legally proscribed 
in some contexts. There may, however, be 
legal requirements to consider alternative 
selection procedures in some such situations. 

There is often tension between the per¬ 
spective that equates fairness with lack of 
bias, in the technical sense, and the perspec¬ 
tive that focuses on testing outcomes. A test 
that is valid for its intended purpose might be 
considered fair if a given test score predicts 
the same performance level for members of 
all groups. It might nonetheless be regarded 
by some as unfair, however, if average test 
scores differ across groups. This is because a 
given selection score and criterion threshold 
will often result in proportionately more false 
negative decisions in groups with lower mean 
test scores. In other words, a lower-scoring 
group will usually have a higher proportion 
of examinees who are rejected on the basis 
of their test scores even though they would 
have performed successfully if they had been 
selected. This seeming paradox is a statistical 
consequence of the imperfect correlation 
between test and criterion. It does not occur 
because of any other property of the test and 
has no direct relationship to group demo¬ 
graphics. It is a purely statistical phenomenon 
that occurs as a function of lower test scores, 
regardless of group membership. For exam¬ 
ple, it usually occurs when the top and bot¬ 
tom test score halves of the majority group 
are compared. The fairness of a test or 
another predictor should be evaluated rela¬ 
tive to that of nontest alternatives that 
might be used instead. 

Group Outcome Differences Due to Choice of 
Predictors 

Success in virtually all real-world 
endeavors requires multiple skills and abili¬ 
ties, which may interact in complex ways. 
Testing programs typically address only a 


79 


JA2321 


AERA APA NOME 0000088 


Case l:14-cv-00857-TSC Document 60-25 
USCA Case #17-7035 Document #1715850 


Filed 12/21/15 Page 90 of 100 
Filed: 01/31/2018 Page 563 of 573 


(standards 


FAIRNESS IN TESTING AND TEST USE / PART II 


subset of these- Some skills and abilities are 
excluded because they are assessed in other 
components of the selection process (e.g., 
completion of course work or an interview); 
others may be excluded because reliable and 
valid measurement is economically, logisti- 
cally, or administratively infeasible. Success 
in college, for example, requires persever¬ 
ance, motivation, good study habits, and a 
host of other factors in addition to verbal 
and quantitative reasoning ability. Even if 
each of the criteria employed in a selection 
process is demonstrably valid and appropri¬ 
ate for that purpose, issues of fairness may 
arise in the choice of which factors are 
measured. If identifiable groups differ in 
their average levels of measured versus 
unmeasured job-relevant characteristics, 
then fairness becomes a concern at the 
group level as well as the individual level. 

Can Consensus Be Achieved? 

It is unlikely that consensus in society at 
large or within the measurement communi¬ 
ty is imminent on all matters of fairness in 
the use of tests. As noted earlier, fairness is 
defined in a variety of ways and is not 
exclusively addressed in technical terms; it is 
subject to different definitions and interpre¬ 
tations in different social and political cir¬ 
cumstances. According to one view, the 
conscientious application of an unbiased 
rest in any given situation is fair, regardless 
of the consequences for individuals or 
groups. Others would argue that fairness 
requires more than satisfying certain techni¬ 
cal requirements. It bears repeating that 
while the Standards will provide more spe¬ 
cific guidance on matters of technical ade¬ 
quacy, matters of values and public policy 
are crucial to responsible test use. 


Standard 7.1 

When credible research reports that test 
scores differ in meaning across examinee 
subgroups for the type of test in question, 
then to the extent feasible, the same forms 
of validity evidence collected for the exam¬ 
inee population as a whole should also be 
collected for each relevant subgroup. 
Subgroups may be found to differ with 
respect to appropriateness of test content, 
internal structure of test responses, the 
relation of test scores to other variables, or 
the response processes employed by indi¬ 
vidual examinees. Any such findings should 
receive due consideration in the interpreta¬ 
tion and use of scores as well as in subse¬ 
quent test revisions. 

Comment: Scores differ in meaning across 
subgroups when the same score produces 
systematically different inferences about 
examinees who are members of different 
subgroups. In those circumstances where 
credible research reports differences in score 
meaning for particular subgroups for the type 
of test in question, this standard calls for 
separate, parallel analyses of data for members 
of those subgroups, sample sizes permitting. 
Relevant examinee subgroups may be defined 
by race or ethnicity, culture, language, gender, 
disability, age, socioeconomic status, or other 
classifications. Not all forms of evidence can 
be examined separately for members of all 
such groups. The validity argument may rely 
on existing research literature, for example, 
and such literature may not be available for 
some populations. For some kinds of evi¬ 
dence, some separate subgroup analyses may 
not be feasible due to the limited number 
of cases available. Data may sometimes be 
accumulated so that these analyses can be 
performed after the test has been in use for a 
period of time. This standard is not satisfied 
by assuring that such groups are represented 
within larger, pooled samples, although this 
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STANDARDS] 


may also be important. In giving “due con¬ 
sideration in the interpretation and use of 
scores,” pursuant to this standard, test users 
should be mindful of legal restrictions that 
may prohibit or limit within-group scoring 
and other practices. 

Standard 7.2 

When credible research reports differences 
in the effects of construct-irrelevant variance 
across subgroups of test takers on perform¬ 
ance on some part of the test, the test 
should be used if at all only for those 
subgroups for which evidence indicates 
that valid inferences can be drawn from 

Comment: An obvious reason why a test 
may not measure the same constructs across 
subgroups is that different components come 
into play from one subgroup to another. 
Alternatively, an irrelevant component may 
have a more significant effect on the perform¬ 
ance of examinees in one subgroup than in 
another. Such intrusive elements are rarely 
entirely absent for any subgroup but are sel¬ 
dom present to any great extent. The decision 
whether or not to use a test with any given 
examinee subgroup necessarily involves a 
careful analysis of the validity evidence for 
different subgroups, as called for in Standard 
7.1, and the exercise of thoughtful profession¬ 
al judgment regarding the significance of the 
irrelevant components. 

A conclusion that a test is not appro¬ 
priate for a particular subgroup requires 
an alternative course of action. This may 
involve a search for a test that can be used 
for all groups or, in circumstances where it 
is feasible to use different construct-equiva¬ 
lent tests for different groups, for an alter¬ 
native test for use in the subgroup for 
which the intended construct is not well 
measured by the current test. In some cases 
multiple tests may be used in combination, 


and a composite that permits valid infer¬ 
ences across subgroups may be identified. 
In some circumstances, such as employment 
testing, there may be legal or other con¬ 
straints on the use of different tests for 
different subgroups. 

It is acknowledged that there are 
occasions where examinees may request or 
demand to take a version of the test other 
than that deemed most appropriate by the 
developer or user. An individual with a 
disability may decline an alternate form 
and request the standard form. Acceding 
.to this request, after ensuring that the 
examinee is fully informed about the test 
and how it will be used, is not a violation 
of this standard. 

Standard 7.3 

When credible research reports that differ¬ 
ential item functioning exists across age, 
gender, racial/ethnic, cultural, disability, 
and/or linguistic groups in the population 
of test takers in the content domain meas¬ 
ured by the test, test developers should 
conduct appropriate studies when feasible. 
Such research should seek to detect and 
eliminate aspects of test design, content, 
and format that might bias test scores for 
particular groups. 

Comment: Differential item functioning 
exists when examinees of equal ability 
differ, on average, according to their group 
membership in their responses to a particu¬ 
lar item. In some domains, existing research 
may indicate that differential item function¬ 
ing occurs infrequently and docs not repli¬ 
cate across samples. In others, research 
evidence may indicate that differential item 
functioning occurs reliably at meaningful 
above-chance levels for some particular 
groups; it is to such circumstances that the 
standard applies. Although it may not be 
possible prior to first release of a test to 
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study the question of differential item 
functioning for some such groups, contin¬ 
ued operational use of a test may afford 
opportunities to check for differential 
item functioning. 

Standard 7.4 

Test developers should strive to identify 
and eliminate language, symbols, words, 
phrases, and content that are generally 
regarded as offensive by members of racial, 
ethnic, gender, or other groups, except 
when judged to be necessary for adequate 
representation of the domain. 

Comment: Two issues are involved. The first 
deals with the inadvertent use of language 
that, unknown to the test developer, has a 
different meaning or connotation in one 
subgroup than in others. Test publishers 
often conduct sensitivity reviews of all test 
material to detect and remove sensitive 
material from the test. The second deals 
with settings in which sensitive material is 
essential for validity. For example, history 
tests may appropriately include material on 
slavery or Nazis. Tests on subjects from the 
life sciences may appropriately include 
material on evolution. A test of under¬ 
standing of an organizations sexual harass¬ 
ment policy may require employees to 
evaluate examples of potentially offensive 
behavior. 

Standard 7.5 

In testing applications involving individu¬ 
alized interpretations of test scores other 
than selection, a test taker’s score should 
not be accepted as a reflection of standing 
on the characteristic being assessed with¬ 
out consideration of alternate explanations 
for the test takers performance on that test 
at that time. 


Comment: Many test manuals point out 
variables that should be considered in inter¬ 
preting test scores, such as clinically relevant 
history, school record, vocational status, and 
test-taker motivation. Influences associated 
with variables such as socioeconomic status, 
ethnicity, gender, cultural background, lan¬ 
guage, or age may also be relevant. In addi¬ 
tion, medication, visual impairments, or 
other disabilities may affect a test taker’s 
performance on, for example, a paper-and- 
pencil test of mathematics. 

Standard 7.6 

When empirical studies of differential pre¬ 
diction of a criterion for members of dif¬ 
ferent subgroups are conducted, they 
should include regression equations (or 
an appropriate equivalent) computed sepa¬ 
rately for each group or treatment under 
consideration or an analysis in which the 
group or treatment variables are entered 
as moderator variables. 

Comment: Correlation coefficients provide 
inadequate evidence for or against a differ¬ 
ential prediction hypochesis if groups or 
treatments are found not to be approxi¬ 
mately equal with respect to both test 
and criterion means and variances. 
Considerations of both regression slopes 
and intercepts are needed. For example, 
despite equal correlations across groups, 
differences in intercepts may be found. 

Standard 7.7 

In testing applications where the level of 
linguistic or reading ability is not part of 
the construct of interest, the linguistic or 
reading demands of the test should be kept 
to the minimum necessary for the valid 
assessment of the intended construct. 
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Comment: When the intent is to assess ability 
in mathematics or mechanical comprehen¬ 
sion, for example, the test should not con¬ 
tain unusual words or complicated syntactic 
conventions unrelated to the mathematical 
or mechanical skill being assessed. 

Standard 7.8 

When scores are disaggregated and pub¬ 
licly reported for groups identified by 
characteristics such as gender, ethnicity, 
age, language proficiency, or disability, 
cautionary statements should be included 
whenever credible research reports that test 
scores may not have comparable meaning 
across these different groups. 

Comment: Comparisons across groups are 
only meaningful if scores have comparable 
meaning across groups. The standard is 
intended as applicable to settings where 
scores are implicidy or explicitly presented as 
comparable in score meaning across groups. 

Standard 7.9 

When tests or assessments are proposed 
for use as instruments of social, education¬ 
al, or public policy, the test developers or 
users proposing the test should fully and 
accurately inform policymakers of the 
characteristics of the tests as well as any 
relevant and credible information that may 
be available concerning the likely conse- 

Standard 7.10 

When the use of a test results in outcomes 
that affect the life chances or educational 
opportunities of examinees, evidence of 
mean test score differences between rele¬ 
vant subgroups of examinees should, 
where feasible, be examined for subgroups 
for which credible research reports mean 
differences for similar tests. Where mean 


differences are found, an investigation 
should be undertaken to determine that 
such differences are not attributable to a 
source of construct underrepresentation 
or construct-irrelevant variance. While 
initially the responsibility of the test 
developer, the test user bears responsibility 
for uses with groups other than those 
specified by the developer. 

Comment: Examples of such test uses 
include situations in which a test plays a 
dominant role in a decision to grant or 
withhold a high school diploma or to pro¬ 
mote a student or retain a student in grade. 
Such an investigation might include a 
review of the cumulative research literature 
or local studies, as appropriate. In some 
domains, such as cognitive ability testing 
in employment, a substantial relevant 
research base may preclude the need for 
local studies. In educational settings, as dis¬ 
cussed in chapter 13, potential differences 
in opportunity to learn may be relevant as 
a possible source of mean differences. 

Standard 7.11 

When a construct can be measured in dif¬ 
ferent ways that are approximately equal 
in their degree of construct representation 
and freedom from construct-irrelevant 
variance, evidence of mean score differ¬ 
ences across relevant subgroups of exam¬ 
inees should be considered in deciding 

Comment: Mean score differences, while 
important, are but one factor influencing 
the choice between one test and another. 
Cost, testing time, test security, and logistic 
issues (e.g., an application where very large 
numbers of examinees must be screened in 
a very short time) are among the issues also 
entering into the professional judgment 
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Standard 7.12 

The testing or assessment process should 
be carried out so that test takers receive 
comparable and equitable treatment dur¬ 
ing all phases of the testing or assessment 

Comment: For example, should a person 
administering a test or interpreting test 
results recognize a personal bias for or 
against an examinee, or for or against any 
subgroup of which the examinee is a mem¬ 
ber, the person could take a variety of steps 
ranging from seeking a review of test inter¬ 
pretations from a colleague to withdrawal 
from the testing process. 


JA2326 


AERA APA NOME 0000093 



Case l:14-cv-00857-TSC Document 60-25 Filed 12/21/15 Page 95 of 100 
USCA Case #17-7035 Document #1715850 Filed: 01/31/2018 Page 568 of 573 


8. THE RIGHTS AND RESPONSIBILITIES 
OF TEST TAKERS 


Background 

This chapter addresses fairness issues unique 
to the interests of the individual test taker. 
Fair treatment of rest takers is not only a mat¬ 
ter of equity, but also promotes the validity 
and reliability of the inferences made from 
the test performance. The standards presented 
in this chapter reflect widely accepted princi¬ 
ples in the field of measurement. The stan¬ 
dards address the responsibilities of test takers 
with regard to test security, their access to test 
results, and their rights when irregularities in 
their testing are claimed. Other issues of fair¬ 
ness are treated in other chapters: general 
principles in chapter 7: the testing of linguis¬ 
tic minorities in chapter 9; the testing of per¬ 
sons with disabilities in chapter 10. General 
considerations concerning reports of test 
results are covered in chapter 5. 

Test takers have the right to be assessed 
with tests that meet current professional stan¬ 
dards, including standards of technical quali¬ 
ty, fairness, administration, and reporting of 
results. Fair and equitable treatment of test 
takers involves providing, in advance of test¬ 
ing, information about the nature of the test, 
the intended use of test scores, and the confi¬ 
dentiality of the results. Test takers, or their 
legal representatives when appropriate, need 
enough information about the test and the 
intended use of cesc results to reach a compe¬ 
tent decision about participating in testing. 
In some instances, formal informed consent 
for testing is required by law or by other stan¬ 
dards of professional practice, such as those 
governing research on human subjects. The 
greater the consequences to the test taker, 
the greater the importance of ensuring that 
the test taker is fully informed about the test 
and voluntarily consents to participate, 
except when testing without consent is per¬ 
mitted by law. If a test is optional, the test 


taker has the right to know the consequences 
of taking or not taking the test. The test 
taker has the right to acceptable opportuni¬ 
ties for asking questions or expressing con¬ 
cerns, and may expect timely responses to 
legitimate questions. 

Where consistent with the purposes 
and nature of the assessment, general infor¬ 
mation is usually provided about the test’s 
content and purposes. Some programs, in 
the interests of fairness, provide all test tak¬ 
ers with helpful materials, such as study 
guides, sample questions, or complete sam¬ 
ple tests, when such information does not 
jeopardize the validity of the results from 
future test administration. Advice may also 
be provided about test-taking strategies, 
including time management, and the advis¬ 
ability of omitting an item response, when 
it is permitted. Information is made known 
about the availability of special accommoda¬ 
tions for those who need them. The policy 
on retesting may be stated, in case the test 
taker feels that the present performance 
does not appropriately reflect his/her best 
performance. 

As participants in the assessment, test 
takers have responsibilities as well as rights. 
Their responsibilities include preparing them¬ 
selves for the test, following the directions of 
the test administrator, representing them¬ 
selves honestly on the test, and informing 
appropriate persons if they believe the test 
results do not adequately reflect them. In 
group testing situations, test takers are expect¬ 
ed not to interfere with the performance of 
other test takers. 

Test validity rests on the assumption 
that a test taker has earned fairly a particu¬ 
lar score or pass/fail decision. Any form of 
cheating, or other behavior that reduces the 
fairness and validity of a test, is irresponsi- 
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ble, is unfair to ocher test takers and may 
lead to sanctions. It is unfair for a test taker 
to use aids that are prohibited. It is unfair 
for a test taker to arrange for someone else 
to take the test in his/her place. The test taker 
is obligated to respect the copyrights of the 
test publisher or sponsor on all test materials. 
This means that rhe test taker will not repro¬ 
duce the items without authorization nor 
disseminate, in any form, material that is 
dearly analogous to the reproduction of the 
items. Test takers, as well as test administra¬ 
tors, have the responsibility not to compro¬ 
mise security by divulging any details of the 
tesc items to others nor may they request 
such details from others. Failure to honor 
these responsibilities may compromise the 
validity of test score interpretations for 
themselves and for others. 

Sometimes, testing programs use special 
scores, statistical indicators, and other 
indirect information about irregularities in 
testing to help ensure that the test scores 
are obtained fairly. Unusual patterns of 
responses, large changes in test scores upon 
retesting, speed of responding, and similar 
indicators may trigger careful scrutiny of 
certain testing protocols. The details of 
these procedures are generally kept secure 
to avoid compromising their use. However, 
test takers can be made aware that in special 
circumstances, such as response or test score 
anomalies, their test responses may get 
special scrutiny. If evidence of impropriety 
or fraud so warrants, the test taker’s score 
may be canceled, or other action taken. 

Because these Standards are directed 
to test providers, and not to test takers, 
standards about test-taker responsibilities 
are phrased in terms of providing informa¬ 
tion to tesr takers about their rights and 
responsibilities. Providing this information 
is the joint responsibility of the test devel¬ 
oper, the test administrator, the test proctor, 
if any, and the test user and may be appor¬ 
tioned according to particular circumstances. 


Standard 8.1 

Any information about test content and 
purposes that is available to any test taker 
prior to testing should be available to all 
test takers. Important information should 
be available free of charge and in accessi¬ 
ble formats. 

Comment: The intent of this standard is 
equal treatment for all. Important informa¬ 
tion would include that necessary for test¬ 
ing, such as when and where the test is 
given, what material should be brought, 
the purpose of the test, and so forth. More 
detailed information, such as practice mate¬ 
rials, is sometimes offered for a fee. Such 
offerings should be made to all test takers. 

Standard 8.2 

Where appropriate, test takers should be 
provided, in advance, as much information 
about the test, the testing process, the 
intended test use, test scoring criteria, 
testing policy, and confidentiality protec¬ 
tion as is consistent with obtaining valid 
responses. 

Comment: Where appropriate, tesc takers 
should be informed, possibly by a test bul¬ 
letin or similar procedure, about test con¬ 
tent, including subject area, topics covered, 
and item formats. They should be informed 
about che advisability of omitting responses. 
They should be aware of any imposed time 
limits, so that they can manage their time 
appropriately. General advice should be 
given about test-taking strategy. In computer 
administrations, they should be told 
about any provisions for review of items 
they have previously answered or omitted. 
Test takers should understand the intended 
use of test scores and the confidentiality of 
test results. They should be advised whether 
they will have access to their results. They 
should be informed about the policy con- 
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cerning taking the test again and about 
the possibility that some test protocols 
may receive special scrutiny for security 
reasons. Test takers should be informed 
about the consequences of misconduct or 
improper behavior, such as cheating, that 
could result in their being prohibited from 
completing the test, receiving test scores, 
or other sanctions. 

Standard 8.3 

When the test taker is offered a choice of 
test format, information about the charac¬ 
teristics of each format should be provided. 

Comment: Test takers sometimes have to 
choose between a paper-and-pencil admi¬ 
nistration and a computer-administered 
test, which may be adaptive. Some tests 
are offered in several different languages. 
Sometimes an alternative assessment is 
offered in lieu of the ordinary test. Test 
takers need to know the characteristics of 
each alternative so that they can make an 
informed choice. 

Standard 8.4 

Informed consent should be obtained from 
test takers, or their legal representatives 
when appropriate, before testing is done 
except (a) when testing without consent 
is mandated by law or governmental regu¬ 
lation, (b) when testing is conducted as 
a regular part of school activities, or (c) 
when consent is clearly implied. 

Comment: Informed consent implies that 
the test takers or representatives are made 
aware, in language that they can under¬ 
stand, of the reasons for testing, the type 
of tests to be used, the intended use, and 
the range of material consequences of 
the intended use. If written, video, or 
audio records are made of the testing ses¬ 
sion, or other records are kept, test takers 


are entitled to know what testing informa¬ 
tion will be released and to whom. Consent 
is not required when testing is legally man¬ 
dated, such as a court-ordered psychological 
assessment, but there may be legal require¬ 
ments for providing information. When 
testing is required for employment or for 
educational admissions, applicants, by 
applying, have implicitly given consent to 
the testing. Nevertheless, test takers and/ 
or their legal representatives should be 
given appropriate information about a test 
when it is in their interest to be informed. 
Young test takers should receive an explana¬ 
tion of the reasons for testing. Even a child 
as young as two or three, as well as older 
test takers of limited cognitive ability, can 
understand a simple explanation as to why 
they are being tested (such as, “I’m going 
to ask you to try to do some things so 
that I can see what you know how to do 
and what things you could use some more 
help with"). 

Standard 8.5 

Test results identified by the names of 
individual test takers, or by other perso¬ 
nally identifying information, should be 
released only to persons with a legitimate, 
professional interest in the test taker or 
who are covered by the informed consent 
of the test taker or a legal representative, 
unless otherwise required by law. 

Comment: Scores of individuals identified 
by name, or by some other means by which 
a person can be readily identified, such as 
social security number, should be kept con¬ 
fidential. In some situations, information 
may be provided on a confidential basis to 
other practitioners with a legitimate interest 
in the particular case, consistent with legal 
and ethical considerations. Information 
may be provided to researchers if a test 
takers anonymity is maintained and the 
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intended use is consistent with accepted 
research practice and is not inconsistent 
with the conditions of the test taker’s 
informed consent. 

Standard 8.6 

Test data maintained in data files should 
be adequately protected from improper 
disclosure. Use of facsimile transmission, 
computer networks, data banks, and other 
electronic data processing or transmittal 
systems should be restricted to situa¬ 
tions in which confidentiality can be 
reasonably assured. 

Comment: When facsimile or computer 
communication is used to transmit a test 
protocol to another site for scoring, or if 
scores are similarly transmitted, special pro¬ 
visions should be made to keep the infor¬ 
mation confidential. See Standard 5.13. 

Standard 8.7 

Test takers should be made aware that 
having someone else take the test for 
them, disclosing confidential test materi¬ 
al, or any other form of cheating is inap¬ 
propriate and that such behavior may 
result in sanctions. 

Comment: Although the standards cannot 
regulate the behavior of test takers, test 
takers should be made aware of their per¬ 
sonal and legal responsibilities. Arranging 
for someone else to impersonate the nom¬ 
inal test taker constitutes fraud. Disclosure 
of confidential testing material for the pur¬ 
pose of giving other test takers pre-knowl¬ 
edge is unfair and may constitute copyright 
infringement. In licensure and certification 
tests, such actions may compromise public 
health and safety. The validity of test score 
interpretations is compromised by inappro¬ 
priate test disclosure. 


Standard 8.8 

When score reporting includes assigning 
individuals to categories, the categories 
should be chosen carefully and described 
precisely. The least stigmatizing labels, 
consistent with accurate representation, 
should always be assigned. 

Comment: When labels are associated with 
tesc results, care should be taken to be pre¬ 
cise in the meanings associated with the 
labels and co avoid unnecessarily stigmatiz¬ 
ing consequences associated with a label. 
For example, in an assessment designed to 
aid in determining whether an individual is 
competent to stand trial, the label “incom¬ 
petent” is appropriate for individuals who 
perform poorly on the assessment. However, 
in a test of basic literacy skills, it is more 
appropriate to use a label such as “not pro¬ 
ficient” rather than “incompetent," because 
the latter term has a more global and 
derogatory meaning. 

Standard 8.9 

When test scores are used to make deci¬ 
sions about a test taker or to make recom¬ 
mendations to a test taker or a third party, 
the test taker or the legal representative is 
entitled to obtain a copy of any report of 
test scores or test interpretation, unless 
that right has been waived or is prohibited 
by law or court order. 

Comment: In some cases a test taker may be 
adequately informed when the test report is 
given to an appropriate third party (treating 
psychologist or psychiatrist) who can inter¬ 
pret the findings to the test taker. In profes¬ 
sional applications of individualized testing, 
when the test taker is given a copy of the 
test report, the examiner or a knowledgeable 
third party should be available to interpret 
it, even if it is clearly written, as the test 
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taker may misunderstand or raise questions 
not specifically answered in the report. In 
employment testing situations, where test 
results are used solely for the purpose of 
aiding selection decisions, waivers of access 
are often a condition of employment, 
although access to test information may 
often be appropriately required in ocher 
circumstances. 

Standard 8.10 

In educational testing programs and in 
licensing and certification applications, 
when an individual score report is expected 
to be delayed beyond a brief investigative 
period, because of possible irregularities 
such as suspected misconduct, the test 
taker should be notified, the reason given, 
and reasonable efforts made to expedite 
review and to protect the interests of the 
test taker. The test taker should be noti¬ 
fied of the disposition, when the investi¬ 
gation is closed. 

Standard 8.11 

In educational testing programs and in 
licensing and certification applications, 
when it is deemed necessary to cancel or 
withhold a test taker’s score because of pos¬ 
sible testing irregularities, including sus¬ 
pected misconduct, the type of evidence 
and procedures to be used to investigate 
the irregularity should be explained to all 
test takers whose scores are directly affected 
by the decision. Test takers should be given 
a timely opportunity to provide evidence 
that the score should not be canceled or 
withheld. Evidence considered in deciding 
upon the final action should be made avail¬ 
able to the test taker on request. 

Comment: Any form of cheating or behavior 
that reduces the validity and fairness of test 
results should be investigated promptly, and 


appropriate action taken. Withholding or 
canceling a test score may arise because of 
suspected misconduct by the test taker, or 
because of some anomaly involving others, 
such as theft, or administrative mishap. An 
avenue of appeal should be available and 
made known to candidates whose scores 
may be amended or withheld. Some testing 
organizations offer the option of a prompt 
and free retest or arbitration of disputes. 

Standard 8.12 

In educational testing programs and in 
licensing and certification applications, 
when testing irregularities are suspected, 
reasonably available information bearing 
directly on the assessment should be con¬ 
sidered, consistent with the need to pro¬ 
tect the privacy of test takers. 

Comment: Unless allegations of misconduct 
are made by associates of the test taker, the 
information to be collected would ordinari¬ 
ly be limited to that obtainable without 
invading the privacy of the test taker or 
his/her associates. 

Standard 8.13 

In educational testing programs and in 
licensing and certification applications, 
test takers are entitled to fair considera¬ 
tion and reasonable process, as appropriate 
to the particular circumstances, in resolv¬ 
ing disputes about testing. Test takers are 
entitled to be informed of any available 
means of recourse. 

Comment: When a test taker’s score may 
be questioned and may be invalidated, or 
when a test taker seeks a review or revision 
of his/her score or some other aspect of the 
testing, scoring, or reporting process, the 
test taker is entided to some orderly process 
for effective input into or review of the 
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decision making of the test administrator or 
test user. Depending upon the magnitude of 
the consequences associated with the test, 
this can range from an internal review of all 
relevant data by a test administrator, to an 
informal conversation with an examinee, to 
a full administrative hearing. The greater 
the consequences, the greater the extent of 
procedural protections that should be made 
available. Test takers should also be made 
aware of procedures for recourse, fees, 
expected time for resolution, and any possi¬ 
ble consequences for the test taker. Some 
testing programs advise that the test taker 
may be represented by an attorney, although 
possibly at the test takers expense. 
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